Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250624 07:57:22.093112 18287 test_util.cc:276] Using random seed: -1381808195
W20250624 07:57:23.304718 18287 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.168s user 0.451s sys 0.715s
W20250624 07:57:23.305024 18287 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.168s user 0.451s sys 0.715s
I20250624 07:57:23.306979 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:57:23.307148 18287 ts_itest-base.cc:116] --------------
I20250624 07:57:23.307298 18287 ts_itest-base.cc:117] 4 tablet servers
I20250624 07:57:23.307458 18287 ts_itest-base.cc:118] 3 replicas per TS
I20250624 07:57:23.307606 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:57:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:57:23Z Disabled control of system clock
I20250624 07:57:23.345050 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:40117
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:40117 with env {}
W20250624 07:57:23.637023 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:23.637558 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:23.637965 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:23.668687 18301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:57:23.668968 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:23.669188 18301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:57:23.669378 18301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:57:23.704201 18301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:40117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:23.705498 18301 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:23.707137 18301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:23.720999 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:23.721100 18310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:23.722339 18308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:23.723474 18301 server_base.cc:1048] running on GCE node
I20250624 07:57:24.900836 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:24.903360 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:24.904733 18301 hybrid_clock.cc:648] HybridClock initialized: now 1750751844904700 us; error 51 us; skew 500 ppm
I20250624 07:57:24.905537 18301 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:24.911537 18301 webserver.cc:469] Webserver started at http://127.17.219.254:33593/ using document root <none> and password file <none>
I20250624 07:57:24.912451 18301 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:24.912665 18301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:24.913111 18301 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:24.917611 18301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7977cfb408fa411b923fb426f18e1843"
format_stamp: "Formatted at 2025-06-24 07:57:24 on dist-test-slave-qhgl"
I20250624 07:57:24.918735 18301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7977cfb408fa411b923fb426f18e1843"
format_stamp: "Formatted at 2025-06-24 07:57:24 on dist-test-slave-qhgl"
I20250624 07:57:24.925765 18301 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250624 07:57:24.931403 18317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:24.932310 18301 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250624 07:57:24.932672 18301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7977cfb408fa411b923fb426f18e1843"
format_stamp: "Formatted at 2025-06-24 07:57:24 on dist-test-slave-qhgl"
I20250624 07:57:24.933028 18301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:24.991520 18301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:24.993009 18301 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:24.993458 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:25.061612 18301 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:40117
I20250624 07:57:25.061712 18368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:40117 every 8 connection(s)
I20250624 07:57:25.064333 18301 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:57:25.069618 18369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:25.071548 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 18301
I20250624 07:57:25.072073 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:57:25.089859 18369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Bootstrap starting.
I20250624 07:57:25.094903 18369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:25.097077 18369 log.cc:826] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:25.102214 18369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: No bootstrap required, opened a new log
I20250624 07:57:25.122210 18369 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:25.122956 18369 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:57:25.123183 18369 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7977cfb408fa411b923fb426f18e1843, State: Initialized, Role: FOLLOWER
I20250624 07:57:25.123894 18369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:25.124378 18369 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:57:25.124648 18369 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:57:25.124920 18369 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:25.128818 18369 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:25.129459 18369 leader_election.cc:304] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7977cfb408fa411b923fb426f18e1843; no voters:
I20250624 07:57:25.131249 18369 leader_election.cc:290] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:57:25.131994 18374 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:57:25.134161 18374 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 LEADER]: Becoming Leader. State: Replica: 7977cfb408fa411b923fb426f18e1843, State: Running, Role: LEADER
I20250624 07:57:25.135013 18374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:25.136246 18369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:57:25.146102 18376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7977cfb408fa411b923fb426f18e1843. Latest consensus state: current_term: 1 leader_uuid: "7977cfb408fa411b923fb426f18e1843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } } }
I20250624 07:57:25.147130 18376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:25.146997 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7977cfb408fa411b923fb426f18e1843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } } }
I20250624 07:57:25.147841 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:25.151696 18382 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:57:25.161827 18382 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:57:25.179423 18382 catalog_manager.cc:1349] Generated new cluster ID: d47bb241a5744533b73a4b8ffa3ac3ba
I20250624 07:57:25.179683 18382 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:57:25.196784 18382 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:57:25.198751 18382 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:57:25.219606 18382 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Generated new TSK 0
I20250624 07:57:25.220458 18382 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:57:25.237047 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--builtin_ntp_servers=127.17.219.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 07:57:25.555760 18393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:25.556282 18393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:25.556787 18393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:25.588531 18393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:25.589509 18393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:57:25.626055 18393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:25.627414 18393 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:25.629019 18393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:25.647547 18400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:25.649073 18399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:25.648384 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:25.648437 18393 server_base.cc:1048] running on GCE node
I20250624 07:57:26.826597 18393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:26.829330 18393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:26.830790 18393 hybrid_clock.cc:648] HybridClock initialized: now 1750751846830726 us; error 76 us; skew 500 ppm
I20250624 07:57:26.831610 18393 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:26.843835 18393 webserver.cc:469] Webserver started at http://127.17.219.193:44135/ using document root <none> and password file <none>
I20250624 07:57:26.844799 18393 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:26.845009 18393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:26.845468 18393 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:26.850382 18393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "13d94f2ffc4f4e3f8202e605dbe22083"
format_stamp: "Formatted at 2025-06-24 07:57:26 on dist-test-slave-qhgl"
I20250624 07:57:26.851539 18393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "13d94f2ffc4f4e3f8202e605dbe22083"
format_stamp: "Formatted at 2025-06-24 07:57:26 on dist-test-slave-qhgl"
I20250624 07:57:26.859189 18393 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250624 07:57:26.865160 18409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:26.866243 18393 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250624 07:57:26.866583 18393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "13d94f2ffc4f4e3f8202e605dbe22083"
format_stamp: "Formatted at 2025-06-24 07:57:26 on dist-test-slave-qhgl"
I20250624 07:57:26.866981 18393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:26.931954 18393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:26.933426 18393 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:26.933857 18393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:26.936362 18393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:26.940544 18393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:26.940749 18393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:26.941011 18393 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:26.941174 18393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:27.076323 18393 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:44603
I20250624 07:57:27.076422 18521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:44603 every 8 connection(s)
I20250624 07:57:27.078903 18393 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:57:27.081911 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 18393
I20250624 07:57:27.082412 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:57:27.089448 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--builtin_ntp_servers=127.17.219.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:57:27.100355 18522 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:27.100905 18522 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:27.102308 18522 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:27.105460 18334 ts_manager.cc:194] Registered new tserver with Master: 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193:44603)
I20250624 07:57:27.107547 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:48577
W20250624 07:57:27.388880 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:27.389380 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:27.389882 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:27.420621 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:27.421483 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:57:27.456051 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:27.457389 18526 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:27.459193 18526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250624 07:57:28.111691 18522 heartbeater.cc:499] Master 127.17.219.254:40117 was elected leader, sending a full tablet report...
W20250624 07:57:27.476212 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:27.478514 18526 server_base.cc:1048] running on GCE node
W20250624 07:57:27.476174 18532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:27.476979 18533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:28.643690 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:28.645867 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:28.647287 18526 hybrid_clock.cc:648] HybridClock initialized: now 1750751848647240 us; error 63 us; skew 500 ppm
I20250624 07:57:28.648110 18526 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:28.654991 18526 webserver.cc:469] Webserver started at http://127.17.219.194:36625/ using document root <none> and password file <none>
I20250624 07:57:28.655923 18526 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:28.656131 18526 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:28.656592 18526 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:28.661010 18526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "08097c7dc8c647319abd3fbb4b16bd0a"
format_stamp: "Formatted at 2025-06-24 07:57:28 on dist-test-slave-qhgl"
I20250624 07:57:28.662169 18526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "08097c7dc8c647319abd3fbb4b16bd0a"
format_stamp: "Formatted at 2025-06-24 07:57:28 on dist-test-slave-qhgl"
I20250624 07:57:28.669468 18526 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250624 07:57:28.675310 18542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:28.676391 18526 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 07:57:28.676667 18526 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "08097c7dc8c647319abd3fbb4b16bd0a"
format_stamp: "Formatted at 2025-06-24 07:57:28 on dist-test-slave-qhgl"
I20250624 07:57:28.676959 18526 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:28.731192 18526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:28.732646 18526 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:28.733088 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:28.735548 18526 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:28.740001 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:28.740232 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:28.740491 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:28.740648 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:28.876368 18526 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:39875
I20250624 07:57:28.876492 18654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:39875 every 8 connection(s)
I20250624 07:57:28.879022 18526 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:57:28.881798 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 18526
I20250624 07:57:28.882318 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 07:57:28.889200 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--builtin_ntp_servers=127.17.219.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:57:28.902592 18655 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:28.903152 18655 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:28.904534 18655 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:28.907187 18334 ts_manager.cc:194] Registered new tserver with Master: 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875)
I20250624 07:57:28.909096 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:40555
W20250624 07:57:29.211071 18659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:29.211762 18659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:29.212663 18659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:29.271600 18659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:29.273219 18659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:57:29.310752 18659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:29.312054 18659 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:29.313642 18659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:29.330030 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:29.913321 18655 heartbeater.cc:499] Master 127.17.219.254:40117 was elected leader, sending a full tablet report...
W20250624 07:57:29.330376 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:29.330088 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:30.537927 18667 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1205 milliseconds
I20250624 07:57:30.538079 18659 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:57:30.539768 18659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:30.542919 18659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:30.544345 18659 hybrid_clock.cc:648] HybridClock initialized: now 1750751850544301 us; error 48 us; skew 500 ppm
I20250624 07:57:30.545300 18659 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:30.554208 18659 webserver.cc:469] Webserver started at http://127.17.219.195:33387/ using document root <none> and password file <none>
I20250624 07:57:30.555821 18659 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:30.556164 18659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:30.556877 18659 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:30.564905 18659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
format_stamp: "Formatted at 2025-06-24 07:57:30 on dist-test-slave-qhgl"
I20250624 07:57:30.566742 18659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
format_stamp: "Formatted at 2025-06-24 07:57:30 on dist-test-slave-qhgl"
I20250624 07:57:30.574163 18659 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250624 07:57:30.580278 18675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:30.581391 18659 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250624 07:57:30.581703 18659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
format_stamp: "Formatted at 2025-06-24 07:57:30 on dist-test-slave-qhgl"
I20250624 07:57:30.582017 18659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:30.636821 18659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:30.638312 18659 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:30.638756 18659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:30.641202 18659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:30.645171 18659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:30.645370 18659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:30.645565 18659 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:30.645714 18659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:30.783210 18659 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:35415
I20250624 07:57:30.783313 18787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:35415 every 8 connection(s)
I20250624 07:57:30.785799 18659 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:57:30.793782 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 18659
I20250624 07:57:30.794287 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 07:57:30.801452 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.196:0
--local_ip_for_outbound_sockets=127.17.219.196
--webserver_interface=127.17.219.196
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--builtin_ntp_servers=127.17.219.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:57:30.811338 18788 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:30.811762 18788 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:30.813331 18788 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:30.815541 18334 ts_manager.cc:194] Registered new tserver with Master: ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415)
I20250624 07:57:30.816861 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:43143
W20250624 07:57:31.123030 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:31.123551 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:31.124039 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:31.155647 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:31.156510 18792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.196
I20250624 07:57:31.191038 18792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.219.196
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.196
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:31.192340 18792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:31.193913 18792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:31.210979 18801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:31.820138 18788 heartbeater.cc:499] Master 127.17.219.254:40117 was elected leader, sending a full tablet report...
W20250624 07:57:31.211030 18799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:31.211198 18798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:32.470572 18800 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1258 milliseconds
I20250624 07:57:32.470691 18792 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:57:32.471913 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:32.474644 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:32.476114 18792 hybrid_clock.cc:648] HybridClock initialized: now 1750751852476069 us; error 61 us; skew 500 ppm
I20250624 07:57:32.477079 18792 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:32.488874 18792 webserver.cc:469] Webserver started at http://127.17.219.196:45755/ using document root <none> and password file <none>
I20250624 07:57:32.489782 18792 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:32.489974 18792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:32.490386 18792 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:32.494791 18792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
format_stamp: "Formatted at 2025-06-24 07:57:32 on dist-test-slave-qhgl"
I20250624 07:57:32.495849 18792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
format_stamp: "Formatted at 2025-06-24 07:57:32 on dist-test-slave-qhgl"
I20250624 07:57:32.503077 18792 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 07:57:32.509244 18809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:32.510268 18792 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250624 07:57:32.510596 18792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
format_stamp: "Formatted at 2025-06-24 07:57:32 on dist-test-slave-qhgl"
I20250624 07:57:32.510975 18792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:32.558182 18792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:32.560474 18792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:32.561066 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:32.564013 18792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:32.568619 18792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:32.568847 18792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:32.569092 18792 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:32.569247 18792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:32.730988 18792 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.196:40561
I20250624 07:57:32.731078 18921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.196:40561 every 8 connection(s)
I20250624 07:57:32.733594 18792 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250624 07:57:32.740907 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 18792
I20250624 07:57:32.741487 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250624 07:57:32.756958 18922 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:32.757376 18922 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:32.758342 18922 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:32.760535 18334 ts_manager.cc:194] Registered new tserver with Master: 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
I20250624 07:57:32.762018 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.196:49329
I20250624 07:57:32.762332 18287 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250624 07:57:32.807168 18334 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40276:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 07:57:32.889211 18723 tablet_service.cc:1468] Processing CreateTablet for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d (DEFAULT_TABLE table=TestTable [id=ae7b3a1f8d3d47e3a102ee0928418eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:57:32.891419 18723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:32.892656 18590 tablet_service.cc:1468] Processing CreateTablet for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d (DEFAULT_TABLE table=TestTable [id=ae7b3a1f8d3d47e3a102ee0928418eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:57:32.894696 18590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:32.904874 18857 tablet_service.cc:1468] Processing CreateTablet for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d (DEFAULT_TABLE table=TestTable [id=ae7b3a1f8d3d47e3a102ee0928418eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:57:32.907763 18857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:32.922307 18941 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Bootstrap starting.
I20250624 07:57:32.926158 18942 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: Bootstrap starting.
I20250624 07:57:32.934553 18942 tablet_bootstrap.cc:654] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:32.934784 18941 tablet_bootstrap.cc:654] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:32.937144 18941 log.cc:826] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:32.937283 18942 log.cc:826] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:32.937217 18943 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: Bootstrap starting.
I20250624 07:57:32.944269 18941 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: No bootstrap required, opened a new log
I20250624 07:57:32.944442 18942 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: No bootstrap required, opened a new log
I20250624 07:57:32.944798 18941 ts_tablet_manager.cc:1397] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20250624 07:57:32.944960 18942 ts_tablet_manager.cc:1397] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.004s
I20250624 07:57:32.946080 18943 tablet_bootstrap.cc:654] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:32.949013 18943 log.cc:826] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:32.955964 18943 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: No bootstrap required, opened a new log
I20250624 07:57:32.956519 18943 ts_tablet_manager.cc:1397] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20250624 07:57:32.966099 18942 raft_consensus.cc:357] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:32.966959 18942 raft_consensus.cc:383] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:57:32.967217 18942 raft_consensus.cc:738] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08097c7dc8c647319abd3fbb4b16bd0a, State: Initialized, Role: FOLLOWER
I20250624 07:57:32.968091 18942 consensus_queue.cc:260] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:32.971987 18942 ts_tablet_manager.cc:1428] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: Time spent starting tablet: real 0.027s user 0.026s sys 0.000s
I20250624 07:57:32.973827 18941 raft_consensus.cc:357] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:32.974776 18941 raft_consensus.cc:383] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:57:32.975075 18941 raft_consensus.cc:738] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0c25889e8647b89ff6d7b9286c9ec0, State: Initialized, Role: FOLLOWER
I20250624 07:57:32.976002 18941 consensus_queue.cc:260] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:32.980989 18941 ts_tablet_manager.cc:1428] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Time spent starting tablet: real 0.036s user 0.028s sys 0.007s
I20250624 07:57:32.982534 18943 raft_consensus.cc:357] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:32.983469 18943 raft_consensus.cc:383] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:57:32.983783 18943 raft_consensus.cc:738] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a876990b85a46f3a2393e4ec8f2d4c1, State: Initialized, Role: FOLLOWER
I20250624 07:57:32.984640 18943 consensus_queue.cc:260] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:32.987553 18922 heartbeater.cc:499] Master 127.17.219.254:40117 was elected leader, sending a full tablet report...
I20250624 07:57:32.990180 18943 ts_tablet_manager.cc:1428] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: Time spent starting tablet: real 0.033s user 0.033s sys 0.000s
I20250624 07:57:33.026163 18947 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:57:33.026640 18947 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:33.029053 18947 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415), 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
I20250624 07:57:33.041427 18877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" is_pre_election: true
I20250624 07:57:33.041584 18743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" is_pre_election: true
I20250624 07:57:33.042143 18877 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08097c7dc8c647319abd3fbb4b16bd0a in term 0.
I20250624 07:57:33.042254 18743 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08097c7dc8c647319abd3fbb4b16bd0a in term 0.
W20250624 07:57:33.043210 18789 tablet.cc:2378] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:57:33.044021 18545 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08097c7dc8c647319abd3fbb4b16bd0a, ce0c25889e8647b89ff6d7b9286c9ec0; no voters:
I20250624 07:57:33.044814 18947 raft_consensus.cc:2802] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:57:33.045145 18947 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:57:33.045356 18947 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:33.049485 18947 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:33.051515 18947 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [CANDIDATE]: Term 1 election: Requested vote from peers ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415), 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
I20250624 07:57:33.051762 18743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
I20250624 07:57:33.052253 18743 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:33.052490 18877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
I20250624 07:57:33.052914 18877 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:33.056615 18743 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08097c7dc8c647319abd3fbb4b16bd0a in term 1.
I20250624 07:57:33.057541 18545 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08097c7dc8c647319abd3fbb4b16bd0a, ce0c25889e8647b89ff6d7b9286c9ec0; no voters:
I20250624 07:57:33.058154 18947 raft_consensus.cc:2802] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:57:33.058251 18877 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08097c7dc8c647319abd3fbb4b16bd0a in term 1.
I20250624 07:57:33.060284 18947 raft_consensus.cc:695] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [term 1 LEADER]: Becoming Leader. State: Replica: 08097c7dc8c647319abd3fbb4b16bd0a, State: Running, Role: LEADER
I20250624 07:57:33.061225 18947 consensus_queue.cc:237] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:33.073900 18333 catalog_manager.cc:5582] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194). New cstate: current_term: 1 leader_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250624 07:57:33.107157 18287 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250624 07:57:33.110970 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 08097c7dc8c647319abd3fbb4b16bd0a to finish bootstrapping
I20250624 07:57:33.125382 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce0c25889e8647b89ff6d7b9286c9ec0 to finish bootstrapping
I20250624 07:57:33.137610 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3a876990b85a46f3a2393e4ec8f2d4c1 to finish bootstrapping
W20250624 07:57:33.139081 18656 tablet.cc:2378] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:57:33.148805 18287 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250624 07:57:33.151873 18287 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
rpc_addresses {
host: "127.17.219.195"
port: 35415
}
}
ts_infos {
permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a"
rpc_addresses {
host: "127.17.219.194"
port: 39875
}
}
ts_infos {
permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
rpc_addresses {
host: "127.17.219.196"
port: 40561
}
}
W20250624 07:57:33.239128 18923 tablet.cc:2378] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:57:33.523617 18947 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:57:33.540553 18952 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 08097c7dc8c647319abd3fbb4b16bd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 07:57:33.548449 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 18526
W20250624 07:57:33.577690 18811 connection.cc:537] server connection from 127.17.219.194:41499 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 07:57:33.578349 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 18301
I20250624 07:57:33.604629 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:40117
--webserver_interface=127.17.219.254
--webserver_port=33593
--builtin_ntp_servers=127.17.219.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:40117 with env {}
W20250624 07:57:33.904976 18965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:33.905506 18965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:33.905910 18965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:33.939767 18965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:57:33.940048 18965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:33.940248 18965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:57:33.940438 18965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:57:33.977041 18965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:40117
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=33593
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:33.978366 18965 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:33.980060 18965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:33.993636 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:34.140304 18522 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:40117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:40117: connect: Connection refused (error 111)
W20250624 07:57:34.557253 18788 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:40117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:40117: connect: Connection refused (error 111)
W20250624 07:57:34.574518 18922 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:40117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:40117: connect: Connection refused (error 111)
I20250624 07:57:35.037225 18980 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08097c7dc8c647319abd3fbb4b16bd0a)
I20250624 07:57:35.037624 18980 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:35.039824 18980 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875), 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
W20250624 07:57:35.041877 18676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111)
W20250624 07:57:35.057663 18676 leader_election.cc:336] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875): Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111)
I20250624 07:57:35.058321 18984 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08097c7dc8c647319abd3fbb4b16bd0a)
I20250624 07:57:35.058854 18984 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:35.062112 18877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" is_pre_election: true
I20250624 07:57:35.062664 18877 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0c25889e8647b89ff6d7b9286c9ec0 in term 1.
I20250624 07:57:35.065245 18679 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a876990b85a46f3a2393e4ec8f2d4c1, ce0c25889e8647b89ff6d7b9286c9ec0; no voters: 08097c7dc8c647319abd3fbb4b16bd0a
I20250624 07:57:35.066161 18980 raft_consensus.cc:2802] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 07:57:35.066496 18980 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08097c7dc8c647319abd3fbb4b16bd0a)
I20250624 07:57:35.066906 18980 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:57:35.070264 18984 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415), 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875)
I20250624 07:57:35.073637 18980 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
W20250624 07:57:35.077736 18810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111)
W20250624 07:57:35.079792 18676 leader_election.cc:336] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875): Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111)
I20250624 07:57:35.081115 18980 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 2 election: Requested vote from peers 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875), 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
I20250624 07:57:35.081777 18877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
I20250624 07:57:35.082301 18877 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 1 FOLLOWER]: Advancing to term 2
W20250624 07:57:35.088644 18810 leader_election.cc:336] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875): Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111)
I20250624 07:57:35.088963 18877 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0c25889e8647b89ff6d7b9286c9ec0 in term 2.
I20250624 07:57:35.090138 18679 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a876990b85a46f3a2393e4ec8f2d4c1, ce0c25889e8647b89ff6d7b9286c9ec0; no voters: 08097c7dc8c647319abd3fbb4b16bd0a
I20250624 07:57:35.090826 18980 raft_consensus.cc:2802] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:57:35.092254 18743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" is_pre_election: true
I20250624 07:57:35.092758 18980 raft_consensus.cc:695] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 LEADER]: Becoming Leader. State: Replica: ce0c25889e8647b89ff6d7b9286c9ec0, State: Running, Role: LEADER
I20250624 07:57:35.093709 18980 consensus_queue.cc:237] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:35.100363 18812 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a876990b85a46f3a2393e4ec8f2d4c1; no voters: 08097c7dc8c647319abd3fbb4b16bd0a, ce0c25889e8647b89ff6d7b9286c9ec0
I20250624 07:57:35.101276 18984 raft_consensus.cc:2747] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250624 07:57:33.996026 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:33.997112 18965 server_base.cc:1048] running on GCE node
W20250624 07:57:33.994611 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:35.216709 18965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:35.219267 18965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:35.220628 18965 hybrid_clock.cc:648] HybridClock initialized: now 1750751855220583 us; error 52 us; skew 500 ppm
I20250624 07:57:35.221418 18965 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:35.227576 18965 webserver.cc:469] Webserver started at http://127.17.219.254:33593/ using document root <none> and password file <none>
I20250624 07:57:35.228523 18965 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:35.228756 18965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:35.236415 18965 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250624 07:57:35.240867 18995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:35.241853 18965 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250624 07:57:35.242182 18965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7977cfb408fa411b923fb426f18e1843"
format_stamp: "Formatted at 2025-06-24 07:57:24 on dist-test-slave-qhgl"
I20250624 07:57:35.244095 18965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:35.313761 18965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:35.315972 18965 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:35.316581 18965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:35.388280 18965 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:40117
I20250624 07:57:35.388370 19046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:40117 every 8 connection(s)
I20250624 07:57:35.391029 18965 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:57:35.396852 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 18965
I20250624 07:57:35.397337 18287 kudu-admin-test.cc:735] Forcing unsafe config change on tserver ce0c25889e8647b89ff6d7b9286c9ec0
I20250624 07:57:35.400864 19047 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:57:35.407677 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Bootstrap starting.
I20250624 07:57:35.455116 19047 log.cc:826] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:35.477210 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:57:35.478037 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Bootstrap complete.
I20250624 07:57:35.500654 19047 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:35.502919 19047 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7977cfb408fa411b923fb426f18e1843, State: Initialized, Role: FOLLOWER
I20250624 07:57:35.503688 19047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:35.504180 19047 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:57:35.504446 19047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:57:35.504727 19047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:57:35.510679 19047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:35.511412 19047 leader_election.cc:304] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7977cfb408fa411b923fb426f18e1843; no voters:
I20250624 07:57:35.513602 19047 leader_election.cc:290] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 07:57:35.514006 19051 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:57:35.517463 19051 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [term 2 LEADER]: Becoming Leader. State: Replica: 7977cfb408fa411b923fb426f18e1843, State: Running, Role: LEADER
I20250624 07:57:35.518301 19051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } }
I20250624 07:57:35.519148 19047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:57:35.528959 19053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7977cfb408fa411b923fb426f18e1843. Latest consensus state: current_term: 2 leader_uuid: "7977cfb408fa411b923fb426f18e1843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } } }
I20250624 07:57:35.529704 19053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:35.532307 19052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7977cfb408fa411b923fb426f18e1843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7977cfb408fa411b923fb426f18e1843" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40117 } } }
I20250624 07:57:35.533591 19052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:35.547410 19059 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:57:35.559659 19059 catalog_manager.cc:671] Loaded metadata for table TestTable [id=ae7b3a1f8d3d47e3a102ee0928418eaf]
I20250624 07:57:35.568113 19059 tablet_loader.cc:96] loaded metadata for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d (table TestTable [id=ae7b3a1f8d3d47e3a102ee0928418eaf])
I20250624 07:57:35.569605 19059 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:57:35.574198 19059 catalog_manager.cc:1261] Loaded cluster ID: d47bb241a5744533b73a4b8ffa3ac3ba
I20250624 07:57:35.574555 19059 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:57:35.593214 19059 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:57:35.600369 19059 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7977cfb408fa411b923fb426f18e1843: Loaded TSK: 0
I20250624 07:57:35.602845 19059 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:57:35.610492 18922 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:35.616967 19012 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" instance_seqno: 1750751852694264) as {username='slave'} at 127.17.219.196:57523; Asking this server to re-register.
I20250624 07:57:35.618486 18922 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:35.619185 18922 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:35.622442 19012 ts_manager.cc:194] Registered new tserver with Master: 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
W20250624 07:57:35.675498 18676 consensus_peers.cc:487] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 -> Peer 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875): Couldn't send request to peer 08097c7dc8c647319abd3fbb4b16bd0a. Status: Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:57:35.715472 18877 raft_consensus.cc:1273] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Refusing update from remote peer ce0c25889e8647b89ff6d7b9286c9ec0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 07:57:35.716965 19072 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250624 07:57:35.726019 19012 catalog_manager.cc:5582] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 reported cstate change: term changed from 1 to 2, leader changed from 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194) to ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195). New cstate: current_term: 2 leader_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } }
I20250624 07:57:35.767982 18788 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:35.771771 19011 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" instance_seqno: 1750751850748393) as {username='slave'} at 127.17.219.195:50961; Asking this server to re-register.
I20250624 07:57:35.773382 18788 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:35.773993 18788 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:35.777091 19011 ts_manager.cc:194] Registered new tserver with Master: ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415)
W20250624 07:57:35.815855 19049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:35.816766 19049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:35.852460 19049 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 07:57:36.176546 18522 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40117
I20250624 07:57:36.180065 19011 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" instance_seqno: 1750751847043468) as {username='slave'} at 127.17.219.193:34415; Asking this server to re-register.
I20250624 07:57:36.181996 18522 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:36.182893 18522 heartbeater.cc:507] Master 127.17.219.254:40117 requested a full tablet report, sending...
I20250624 07:57:36.185622 19011 ts_manager.cc:194] Registered new tserver with Master: 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193:44603)
W20250624 07:57:37.302426 19086 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 19049
W20250624 07:57:37.303133 19086 kernel_stack_watchdog.cc:198] Thread 19049 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250624 07:57:37.360801 19049 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.463s user 0.603s sys 0.849s
W20250624 07:57:37.475539 19049 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.578s user 0.609s sys 0.858s
I20250624 07:57:37.521677 18743 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
}
peers {
permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
}
}
from {username='slave'} at 127.0.0.1:60208
W20250624 07:57:37.522735 18743 raft_consensus.cc:2216] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true
I20250624 07:57:37.523568 18743 raft_consensus.cc:3053] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 LEADER]: Stepping down as leader of term 2
I20250624 07:57:37.523792 18743 raft_consensus.cc:738] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ce0c25889e8647b89ff6d7b9286c9ec0, State: Running, Role: LEADER
I20250624 07:57:37.524346 18743 consensus_queue.cc:260] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:37.525190 18743 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:57:38.804622 19105 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce0c25889e8647b89ff6d7b9286c9ec0)
I20250624 07:57:38.805066 19105 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } }
I20250624 07:57:38.806504 19105 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415), 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875)
I20250624 07:57:38.807462 18743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" is_pre_election: true
W20250624 07:57:38.811645 18810 leader_election.cc:336] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194:39875): Network error: Client connection negotiation failed: client connection to 127.17.219.194:39875: connect: Connection refused (error 111)
I20250624 07:57:38.812006 18810 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a876990b85a46f3a2393e4ec8f2d4c1; no voters: 08097c7dc8c647319abd3fbb4b16bd0a, ce0c25889e8647b89ff6d7b9286c9ec0
I20250624 07:57:38.812626 19105 raft_consensus.cc:2747] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250624 07:57:39.031340 19108 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250624 07:57:39.031730 19108 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true
I20250624 07:57:39.032800 19108 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
I20250624 07:57:39.033901 18877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" is_pre_election: true
I20250624 07:57:39.034435 18877 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0c25889e8647b89ff6d7b9286c9ec0 in term 2.
I20250624 07:57:39.035422 18679 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3a876990b85a46f3a2393e4ec8f2d4c1, ce0c25889e8647b89ff6d7b9286c9ec0; no voters:
I20250624 07:57:39.036026 19108 raft_consensus.cc:2802] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250624 07:57:39.036289 19108 raft_consensus.cc:491] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250624 07:57:39.036556 19108 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 3 FOLLOWER]: Advancing to term 4
I20250624 07:57:39.040697 19108 raft_consensus.cc:513] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true
I20250624 07:57:39.041677 19108 leader_election.cc:290] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 4 election: Requested vote from peers 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196:40561)
I20250624 07:57:39.042658 18877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d" candidate_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
I20250624 07:57:39.043084 18877 raft_consensus.cc:3058] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 2 FOLLOWER]: Advancing to term 4
I20250624 07:57:39.047577 18877 raft_consensus.cc:2466] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0c25889e8647b89ff6d7b9286c9ec0 in term 4.
I20250624 07:57:39.048422 18679 leader_election.cc:304] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3a876990b85a46f3a2393e4ec8f2d4c1, ce0c25889e8647b89ff6d7b9286c9ec0; no voters:
I20250624 07:57:39.049046 19108 raft_consensus.cc:2802] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 4 FOLLOWER]: Leader election won for term 4
I20250624 07:57:39.049891 19108 raft_consensus.cc:695] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 4 LEADER]: Becoming Leader. State: Replica: ce0c25889e8647b89ff6d7b9286c9ec0, State: Running, Role: LEADER
I20250624 07:57:39.050565 19108 consensus_queue.cc:237] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true
I20250624 07:57:39.055922 19012 catalog_manager.cc:5582] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195), VOTER 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196). New cstate: current_term: 4 leader_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08097c7dc8c647319abd3fbb4b16bd0a" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 39875 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true }
I20250624 07:57:39.453990 18877 raft_consensus.cc:1273] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Refusing update from remote peer ce0c25889e8647b89ff6d7b9286c9ec0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250624 07:57:39.455564 19108 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250624 07:57:39.463850 19109 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true }
I20250624 07:57:39.464792 18877 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true }
I20250624 07:57:39.476606 19012 catalog_manager.cc:5582] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 reported cstate change: config changed from index -1 to 3, VOTER 08097c7dc8c647319abd3fbb4b16bd0a (127.17.219.194) evicted, no longer has a pending config: VOTER ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195), VOTER 3a876990b85a46f3a2393e4ec8f2d4c1 (127.17.219.196). New cstate: current_term: 4 leader_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } unsafe_config_change: true }
W20250624 07:57:39.484526 19012 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d on TS 08097c7dc8c647319abd3fbb4b16bd0a: Not found: failed to reset TS proxy: Could not find TS for UUID 08097c7dc8c647319abd3fbb4b16bd0a
I20250624 07:57:39.503301 18743 consensus_queue.cc:237] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: true } } unsafe_config_change: true
I20250624 07:57:39.510509 18877 raft_consensus.cc:1273] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Refusing update from remote peer ce0c25889e8647b89ff6d7b9286c9ec0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250624 07:57:39.511819 19109 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250624 07:57:39.517717 19120 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: true } } unsafe_config_change: true }
I20250624 07:57:39.519104 18877 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: true } } unsafe_config_change: true }
W20250624 07:57:39.521802 18679 consensus_peers.cc:487] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 -> Peer 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193:44603): Couldn't send request to peer 13d94f2ffc4f4e3f8202e605dbe22083. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cdaefdcd6724f9d85f9d16cd6cc928d. This is attempt 1: this message will repeat every 5th retry.
I20250624 07:57:39.526878 18998 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250624 07:57:39.529141 19012 catalog_manager.cc:5582] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 reported cstate change: config changed from index 3 to 5, NON_VOTER 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) added. New cstate: current_term: 4 leader_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: true } } unsafe_config_change: true }
W20250624 07:57:39.537014 18997 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d on TS 08097c7dc8c647319abd3fbb4b16bd0a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 08097c7dc8c647319abd3fbb4b16bd0a
I20250624 07:57:39.915045 19126 ts_tablet_manager.cc:927] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Initiating tablet copy from peer ce0c25889e8647b89ff6d7b9286c9ec0 (127.17.219.195:35415)
I20250624 07:57:39.918833 19126 tablet_copy_client.cc:323] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: tablet copy: Beginning tablet copy session from remote peer at address 127.17.219.195:35415
I20250624 07:57:39.931146 18763 tablet_copy_service.cc:140] P ce0c25889e8647b89ff6d7b9286c9ec0: Received BeginTabletCopySession request for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d from peer 13d94f2ffc4f4e3f8202e605dbe22083 ({username='slave'} at 127.17.219.193:49059)
I20250624 07:57:39.931669 18763 tablet_copy_service.cc:161] P ce0c25889e8647b89ff6d7b9286c9ec0: Beginning new tablet copy session on tablet 7cdaefdcd6724f9d85f9d16cd6cc928d from peer 13d94f2ffc4f4e3f8202e605dbe22083 at {username='slave'} at 127.17.219.193:49059: session id = 13d94f2ffc4f4e3f8202e605dbe22083-7cdaefdcd6724f9d85f9d16cd6cc928d
I20250624 07:57:39.937305 18763 tablet_copy_source_session.cc:215] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 07:57:39.942849 19126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:39.961102 19126 tablet_copy_client.cc:806] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: tablet copy: Starting download of 0 data blocks...
I20250624 07:57:39.961613 19126 tablet_copy_client.cc:670] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: tablet copy: Starting download of 1 WAL segments...
I20250624 07:57:39.965601 19126 tablet_copy_client.cc:538] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 07:57:39.971141 19126 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Bootstrap starting.
I20250624 07:57:39.983271 19126 log.cc:826] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:39.994122 19126 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:57:39.994853 19126 tablet_bootstrap.cc:492] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Bootstrap complete.
I20250624 07:57:39.995405 19126 ts_tablet_manager.cc:1397] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20250624 07:57:40.012003 19126 raft_consensus.cc:357] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: true } } unsafe_config_change: true
I20250624 07:57:40.012861 19126 raft_consensus.cc:738] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 13d94f2ffc4f4e3f8202e605dbe22083, State: Initialized, Role: LEARNER
I20250624 07:57:40.013443 19126 consensus_queue.cc:260] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: true } } unsafe_config_change: true
I20250624 07:57:40.017445 19126 ts_tablet_manager.cc:1428] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20250624 07:57:40.019254 18763 tablet_copy_service.cc:342] P ce0c25889e8647b89ff6d7b9286c9ec0: Request end of tablet copy session 13d94f2ffc4f4e3f8202e605dbe22083-7cdaefdcd6724f9d85f9d16cd6cc928d received from {username='slave'} at 127.17.219.193:49059
I20250624 07:57:40.019676 18763 tablet_copy_service.cc:434] P ce0c25889e8647b89ff6d7b9286c9ec0: ending tablet copy session 13d94f2ffc4f4e3f8202e605dbe22083-7cdaefdcd6724f9d85f9d16cd6cc928d on tablet 7cdaefdcd6724f9d85f9d16cd6cc928d with peer 13d94f2ffc4f4e3f8202e605dbe22083
I20250624 07:57:40.378693 18477 raft_consensus.cc:1215] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250624 07:57:40.703526 18997 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 7cdaefdcd6724f9d85f9d16cd6cc928d on TS 08097c7dc8c647319abd3fbb4b16bd0a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 08097c7dc8c647319abd3fbb4b16bd0a
I20250624 07:57:40.983870 19131 raft_consensus.cc:1062] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0: attempting to promote NON_VOTER 13d94f2ffc4f4e3f8202e605dbe22083 to VOTER
I20250624 07:57:40.985428 19131 consensus_queue.cc:237] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: false } } unsafe_config_change: true
I20250624 07:57:40.990165 18477 raft_consensus.cc:1273] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083 [term 4 LEARNER]: Refusing update from remote peer ce0c25889e8647b89ff6d7b9286c9ec0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250624 07:57:40.990690 18877 raft_consensus.cc:1273] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Refusing update from remote peer ce0c25889e8647b89ff6d7b9286c9ec0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250624 07:57:40.991245 19132 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250624 07:57:40.992017 19131 consensus_queue.cc:1035] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250624 07:57:40.997686 19132 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: false } } unsafe_config_change: true }
I20250624 07:57:40.999277 18877 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 3a876990b85a46f3a2393e4ec8f2d4c1 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: false } } unsafe_config_change: true }
I20250624 07:57:41.000113 18477 raft_consensus.cc:2953] T 7cdaefdcd6724f9d85f9d16cd6cc928d P 13d94f2ffc4f4e3f8202e605dbe22083 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: false } } unsafe_config_change: true }
I20250624 07:57:41.007452 19012 catalog_manager.cc:5582] T 7cdaefdcd6724f9d85f9d16cd6cc928d P ce0c25889e8647b89ff6d7b9286c9ec0 reported cstate change: config changed from index 5 to 6, 13d94f2ffc4f4e3f8202e605dbe22083 (127.17.219.193) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44603 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250624 07:57:41.064803 18287 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250624 07:57:41.077196 18287 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "7cdaefdcd6724f9d85f9d16cd6cc928d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "ce0c25889e8647b89ff6d7b9286c9ec0"
rpc_addresses {
host: "127.17.219.195"
port: 35415
}
}
ts_infos {
permanent_uuid: "3a876990b85a46f3a2393e4ec8f2d4c1"
rpc_addresses {
host: "127.17.219.196"
port: 40561
}
}
ts_infos {
permanent_uuid: "13d94f2ffc4f4e3f8202e605dbe22083"
rpc_addresses {
host: "127.17.219.193"
port: 44603
}
}
I20250624 07:57:41.080152 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 18393
I20250624 07:57:41.107300 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 18659
I20250624 07:57:41.134860 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 18792
I20250624 07:57:41.158939 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 18965
2025-06-24T07:57:41Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19128 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250624 07:57:41.219437 18287 test_util.cc:276] Using random seed: -1362681772
I20250624 07:57:41.225024 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:57:41.225175 18287 ts_itest-base.cc:116] --------------
I20250624 07:57:41.225337 18287 ts_itest-base.cc:117] 3 tablet servers
I20250624 07:57:41.225466 18287 ts_itest-base.cc:118] 3 replicas per TS
I20250624 07:57:41.225606 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:57:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:57:41Z Disabled control of system clock
I20250624 07:57:41.260602 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:36805
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:36805
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250624 07:57:41.555462 19153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:41.556015 19153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:41.556417 19153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:41.587193 19153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:57:41.587481 19153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:41.587680 19153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:57:41.587896 19153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:57:41.623056 19153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:36805
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:41.624297 19153 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:41.625826 19153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:41.640416 19159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:41.640512 19160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:41.641534 19162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:42.804752 19161 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:57:42.804844 19153 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:57:42.808593 19153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:42.811179 19153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:42.812508 19153 hybrid_clock.cc:648] HybridClock initialized: now 1750751862812491 us; error 40 us; skew 500 ppm
I20250624 07:57:42.813289 19153 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:42.824370 19153 webserver.cc:469] Webserver started at http://127.17.219.254:32897/ using document root <none> and password file <none>
I20250624 07:57:42.825289 19153 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:42.825487 19153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:42.825954 19153 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:42.830322 19153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a33be3a933f2441d9519a9875cf28325"
format_stamp: "Formatted at 2025-06-24 07:57:42 on dist-test-slave-qhgl"
I20250624 07:57:42.831373 19153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a33be3a933f2441d9519a9875cf28325"
format_stamp: "Formatted at 2025-06-24 07:57:42 on dist-test-slave-qhgl"
I20250624 07:57:42.838542 19153 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250624 07:57:42.844025 19170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:42.845065 19153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250624 07:57:42.845374 19153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a33be3a933f2441d9519a9875cf28325"
format_stamp: "Formatted at 2025-06-24 07:57:42 on dist-test-slave-qhgl"
I20250624 07:57:42.845703 19153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:42.895939 19153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:42.897392 19153 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:42.897830 19153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:42.964738 19153 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:36805
I20250624 07:57:42.964818 19221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:36805 every 8 connection(s)
I20250624 07:57:42.967352 19153 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:57:42.972285 19222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:42.977834 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19153
I20250624 07:57:42.978535 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:57:42.997728 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325: Bootstrap starting.
I20250624 07:57:43.004166 19222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:43.006018 19222 log.cc:826] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:43.010514 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325: No bootstrap required, opened a new log
I20250624 07:57:43.027786 19222 raft_consensus.cc:357] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33be3a933f2441d9519a9875cf28325" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 36805 } }
I20250624 07:57:43.028437 19222 raft_consensus.cc:383] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:57:43.028668 19222 raft_consensus.cc:738] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33be3a933f2441d9519a9875cf28325, State: Initialized, Role: FOLLOWER
I20250624 07:57:43.029309 19222 consensus_queue.cc:260] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33be3a933f2441d9519a9875cf28325" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 36805 } }
I20250624 07:57:43.029809 19222 raft_consensus.cc:397] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:57:43.030064 19222 raft_consensus.cc:491] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:57:43.030350 19222 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:43.034585 19222 raft_consensus.cc:513] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33be3a933f2441d9519a9875cf28325" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 36805 } }
I20250624 07:57:43.035305 19222 leader_election.cc:304] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a33be3a933f2441d9519a9875cf28325; no voters:
I20250624 07:57:43.036897 19222 leader_election.cc:290] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:57:43.037555 19227 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:57:43.039841 19227 raft_consensus.cc:695] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [term 1 LEADER]: Becoming Leader. State: Replica: a33be3a933f2441d9519a9875cf28325, State: Running, Role: LEADER
I20250624 07:57:43.040648 19227 consensus_queue.cc:237] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33be3a933f2441d9519a9875cf28325" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 36805 } }
I20250624 07:57:43.041502 19222 sys_catalog.cc:564] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:57:43.051330 19229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a33be3a933f2441d9519a9875cf28325. Latest consensus state: current_term: 1 leader_uuid: "a33be3a933f2441d9519a9875cf28325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33be3a933f2441d9519a9875cf28325" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 36805 } } }
I20250624 07:57:43.052611 19229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:43.053864 19228 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a33be3a933f2441d9519a9875cf28325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33be3a933f2441d9519a9875cf28325" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 36805 } } }
I20250624 07:57:43.054569 19228 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:43.056633 19234 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:57:43.065989 19234 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:57:43.082908 19234 catalog_manager.cc:1349] Generated new cluster ID: e54175c944364f3198dfd3ca991850ec
I20250624 07:57:43.083153 19234 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:57:43.114390 19234 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:57:43.115841 19234 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:57:43.135699 19234 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a33be3a933f2441d9519a9875cf28325: Generated new TSK 0
I20250624 07:57:43.136636 19234 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:57:43.153211 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:36805
--builtin_ntp_servers=127.17.219.212:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250624 07:57:43.451740 19246 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 07:57:43.452394 19246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:43.452664 19246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:43.453145 19246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:43.484364 19246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:43.485209 19246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:57:43.520292 19246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:36805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:43.521651 19246 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:43.523350 19246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:43.540572 19253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:43.543031 19255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:44.724079 19254 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:57:44.724150 19246 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250624 07:57:43.540693 19252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:44.728854 19246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:44.731559 19246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:44.733081 19246 hybrid_clock.cc:648] HybridClock initialized: now 1750751864733018 us; error 55 us; skew 500 ppm
I20250624 07:57:44.734125 19246 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:44.742095 19246 webserver.cc:469] Webserver started at http://127.17.219.193:40295/ using document root <none> and password file <none>
I20250624 07:57:44.743479 19246 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:44.743753 19246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:44.744313 19246 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:44.750815 19246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
format_stamp: "Formatted at 2025-06-24 07:57:44 on dist-test-slave-qhgl"
I20250624 07:57:44.752275 19246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
format_stamp: "Formatted at 2025-06-24 07:57:44 on dist-test-slave-qhgl"
I20250624 07:57:44.761440 19246 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250624 07:57:44.768803 19262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:44.769953 19246 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 07:57:44.770349 19246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
format_stamp: "Formatted at 2025-06-24 07:57:44 on dist-test-slave-qhgl"
I20250624 07:57:44.770788 19246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:44.832593 19246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:44.834019 19246 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:44.834456 19246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:44.837107 19246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:44.841600 19246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:44.841820 19246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:44.842053 19246 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:44.842212 19246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:44.976404 19246 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:44873
I20250624 07:57:44.976511 19374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:44873 every 8 connection(s)
I20250624 07:57:44.978935 19246 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:57:44.980894 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19246
I20250624 07:57:44.981371 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:57:44.989032 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:36805
--builtin_ntp_servers=127.17.219.212:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 07:57:45.004781 19375 heartbeater.cc:344] Connected to a master server at 127.17.219.254:36805
I20250624 07:57:45.005220 19375 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:45.006279 19375 heartbeater.cc:507] Master 127.17.219.254:36805 requested a full tablet report, sending...
I20250624 07:57:45.008873 19187 ts_manager.cc:194] Registered new tserver with Master: c076d0a5d9dc4a0fb1336b3bfd777057 (127.17.219.193:44873)
I20250624 07:57:45.010824 19187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:43859
W20250624 07:57:45.287204 19379 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 07:57:45.287849 19379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:45.288120 19379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:45.288609 19379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:45.319896 19379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:45.320869 19379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:57:45.355945 19379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:36805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:45.357232 19379 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:45.358873 19379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:45.373337 19386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:46.014111 19375 heartbeater.cc:499] Master 127.17.219.254:36805 was elected leader, sending a full tablet report...
W20250624 07:57:45.373397 19385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:45.375524 19388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:46.519500 19387 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1148 milliseconds
I20250624 07:57:46.519567 19379 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:57:46.524284 19379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:46.527021 19379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:46.528512 19379 hybrid_clock.cc:648] HybridClock initialized: now 1750751866528457 us; error 69 us; skew 500 ppm
I20250624 07:57:46.529302 19379 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:46.536424 19379 webserver.cc:469] Webserver started at http://127.17.219.194:40173/ using document root <none> and password file <none>
I20250624 07:57:46.537325 19379 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:46.537518 19379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:46.537921 19379 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:46.542292 19379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1838aabde50e4fbd959daa364f979069"
format_stamp: "Formatted at 2025-06-24 07:57:46 on dist-test-slave-qhgl"
I20250624 07:57:46.543435 19379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1838aabde50e4fbd959daa364f979069"
format_stamp: "Formatted at 2025-06-24 07:57:46 on dist-test-slave-qhgl"
I20250624 07:57:46.550396 19379 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250624 07:57:46.555852 19395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:46.556850 19379 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250624 07:57:46.557161 19379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1838aabde50e4fbd959daa364f979069"
format_stamp: "Formatted at 2025-06-24 07:57:46 on dist-test-slave-qhgl"
I20250624 07:57:46.557488 19379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:46.621655 19379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:46.623150 19379 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:46.623570 19379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:46.625998 19379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:46.629979 19379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:46.630192 19379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:46.630446 19379 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:46.630601 19379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:46.762233 19379 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:44471
I20250624 07:57:46.762348 19507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:44471 every 8 connection(s)
I20250624 07:57:46.764915 19379 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:57:46.770830 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19379
I20250624 07:57:46.771286 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 07:57:46.777521 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:36805
--builtin_ntp_servers=127.17.219.212:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 07:57:46.786237 19508 heartbeater.cc:344] Connected to a master server at 127.17.219.254:36805
I20250624 07:57:46.786686 19508 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:46.787758 19508 heartbeater.cc:507] Master 127.17.219.254:36805 requested a full tablet report, sending...
I20250624 07:57:46.789847 19187 ts_manager.cc:194] Registered new tserver with Master: 1838aabde50e4fbd959daa364f979069 (127.17.219.194:44471)
I20250624 07:57:46.791101 19187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:40649
W20250624 07:57:47.091684 19512 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 07:57:47.092321 19512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:47.092592 19512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:47.093078 19512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:47.125838 19512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:47.126746 19512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:57:47.164077 19512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:36805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:47.165472 19512 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:47.167152 19512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:47.184608 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:47.794241 19508 heartbeater.cc:499] Master 127.17.219.254:36805 was elected leader, sending a full tablet report...
W20250624 07:57:47.184665 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:47.186683 19512 server_base.cc:1048] running on GCE node
W20250624 07:57:47.185968 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:48.390796 19512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:48.393592 19512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:48.395071 19512 hybrid_clock.cc:648] HybridClock initialized: now 1750751868394984 us; error 94 us; skew 500 ppm
I20250624 07:57:48.396049 19512 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:48.406553 19512 webserver.cc:469] Webserver started at http://127.17.219.195:41657/ using document root <none> and password file <none>
I20250624 07:57:48.407516 19512 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:48.407713 19512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:48.408126 19512 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:48.412523 19512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4f470affd74840b2b87fc9a060873aa3"
format_stamp: "Formatted at 2025-06-24 07:57:48 on dist-test-slave-qhgl"
I20250624 07:57:48.413573 19512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4f470affd74840b2b87fc9a060873aa3"
format_stamp: "Formatted at 2025-06-24 07:57:48 on dist-test-slave-qhgl"
I20250624 07:57:48.421200 19512 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.007s
I20250624 07:57:48.426767 19528 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:48.427753 19512 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250624 07:57:48.428047 19512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4f470affd74840b2b87fc9a060873aa3"
format_stamp: "Formatted at 2025-06-24 07:57:48 on dist-test-slave-qhgl"
I20250624 07:57:48.428349 19512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:48.492771 19512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:48.494220 19512 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:48.494648 19512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:48.497113 19512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:48.501617 19512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:48.501825 19512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:48.502102 19512 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:48.502251 19512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:48.637889 19512 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:36007
I20250624 07:57:48.638002 19640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:36007 every 8 connection(s)
I20250624 07:57:48.640475 19512 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:57:48.641400 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19512
I20250624 07:57:48.642046 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 07:57:48.661242 19641 heartbeater.cc:344] Connected to a master server at 127.17.219.254:36805
I20250624 07:57:48.661778 19641 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:48.662848 19641 heartbeater.cc:507] Master 127.17.219.254:36805 requested a full tablet report, sending...
I20250624 07:57:48.664929 19186 ts_manager.cc:194] Registered new tserver with Master: 4f470affd74840b2b87fc9a060873aa3 (127.17.219.195:36007)
I20250624 07:57:48.666290 19186 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:55509
I20250624 07:57:48.677006 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:57:48.712700 19186 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38468:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 07:57:48.731274 19186 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:57:48.807343 19310 tablet_service.cc:1468] Processing CreateTablet for tablet abee3912c65943d2ae21df2e5d2b2854 (DEFAULT_TABLE table=TestTable [id=a70b5642200a473aa18d139d845b7b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:57:48.807492 19443 tablet_service.cc:1468] Processing CreateTablet for tablet abee3912c65943d2ae21df2e5d2b2854 (DEFAULT_TABLE table=TestTable [id=a70b5642200a473aa18d139d845b7b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:57:48.809135 19310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abee3912c65943d2ae21df2e5d2b2854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:48.809252 19443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abee3912c65943d2ae21df2e5d2b2854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:48.811721 19576 tablet_service.cc:1468] Processing CreateTablet for tablet abee3912c65943d2ae21df2e5d2b2854 (DEFAULT_TABLE table=TestTable [id=a70b5642200a473aa18d139d845b7b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:57:48.813534 19576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abee3912c65943d2ae21df2e5d2b2854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:48.829468 19660 tablet_bootstrap.cc:492] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: Bootstrap starting.
I20250624 07:57:48.836647 19661 tablet_bootstrap.cc:492] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: Bootstrap starting.
I20250624 07:57:48.837438 19660 tablet_bootstrap.cc:654] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:48.839843 19660 log.cc:826] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:48.841210 19662 tablet_bootstrap.cc:492] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: Bootstrap starting.
I20250624 07:57:48.844165 19661 tablet_bootstrap.cc:654] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:48.846148 19660 tablet_bootstrap.cc:492] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: No bootstrap required, opened a new log
I20250624 07:57:48.846442 19661 log.cc:826] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:48.846688 19660 ts_tablet_manager.cc:1397] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.002s
I20250624 07:57:48.847369 19662 tablet_bootstrap.cc:654] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:48.849632 19662 log.cc:826] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:48.852502 19661 tablet_bootstrap.cc:492] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: No bootstrap required, opened a new log
I20250624 07:57:48.853192 19661 ts_tablet_manager.cc:1397] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20250624 07:57:48.856241 19662 tablet_bootstrap.cc:492] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: No bootstrap required, opened a new log
I20250624 07:57:48.856736 19662 ts_tablet_manager.cc:1397] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250624 07:57:48.867457 19660 raft_consensus.cc:357] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:48.868178 19660 raft_consensus.cc:738] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c076d0a5d9dc4a0fb1336b3bfd777057, State: Initialized, Role: FOLLOWER
I20250624 07:57:48.868779 19660 consensus_queue.cc:260] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:48.875089 19660 ts_tablet_manager.cc:1428] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: Time spent starting tablet: real 0.028s user 0.017s sys 0.011s
I20250624 07:57:48.882350 19662 raft_consensus.cc:357] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:48.882274 19661 raft_consensus.cc:357] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:48.883064 19662 raft_consensus.cc:738] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f470affd74840b2b87fc9a060873aa3, State: Initialized, Role: FOLLOWER
I20250624 07:57:48.883185 19661 raft_consensus.cc:738] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1838aabde50e4fbd959daa364f979069, State: Initialized, Role: FOLLOWER
I20250624 07:57:48.883669 19662 consensus_queue.cc:260] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:48.883837 19661 consensus_queue.cc:260] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:48.886814 19641 heartbeater.cc:499] Master 127.17.219.254:36805 was elected leader, sending a full tablet report...
I20250624 07:57:48.887925 19662 ts_tablet_manager.cc:1428] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: Time spent starting tablet: real 0.031s user 0.027s sys 0.005s
I20250624 07:57:48.891721 19661 ts_tablet_manager.cc:1428] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: Time spent starting tablet: real 0.038s user 0.029s sys 0.005s
W20250624 07:57:48.895599 19642 tablet.cc:2378] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:57:48.907296 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:57:48.910495 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c076d0a5d9dc4a0fb1336b3bfd777057 to finish bootstrapping
I20250624 07:57:48.923918 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1838aabde50e4fbd959daa364f979069 to finish bootstrapping
I20250624 07:57:48.934078 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4f470affd74840b2b87fc9a060873aa3 to finish bootstrapping
I20250624 07:57:48.979640 19330 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854"
dest_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
from {username='slave'} at 127.0.0.1:48998
I20250624 07:57:48.980389 19330 raft_consensus.cc:491] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 07:57:48.980779 19330 raft_consensus.cc:3058] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 0 FOLLOWER]: Advancing to term 1
W20250624 07:57:48.988993 19376 tablet.cc:2378] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:57:49.021203 19509 tablet.cc:2378] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:57:49.074105 19330 raft_consensus.cc:513] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:49.078166 19330 leader_election.cc:290] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [CANDIDATE]: Term 1 election: Requested vote from peers 1838aabde50e4fbd959daa364f979069 (127.17.219.194:44471), 4f470affd74840b2b87fc9a060873aa3 (127.17.219.195:36007)
I20250624 07:57:49.085655 18287 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250624 07:57:49.089094 19463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854" candidate_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1838aabde50e4fbd959daa364f979069"
I20250624 07:57:49.089867 19463 raft_consensus.cc:3058] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:49.090571 19596 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854" candidate_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f470affd74840b2b87fc9a060873aa3"
I20250624 07:57:49.091291 19596 raft_consensus.cc:3058] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:49.177116 19463 raft_consensus.cc:2466] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c076d0a5d9dc4a0fb1336b3bfd777057 in term 1.
I20250624 07:57:49.177155 19596 raft_consensus.cc:2466] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c076d0a5d9dc4a0fb1336b3bfd777057 in term 1.
I20250624 07:57:49.178620 19264 leader_election.cc:304] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1838aabde50e4fbd959daa364f979069, c076d0a5d9dc4a0fb1336b3bfd777057; no voters:
I20250624 07:57:49.179863 19666 raft_consensus.cc:2802] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:57:49.182502 19666 raft_consensus.cc:695] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 LEADER]: Becoming Leader. State: Replica: c076d0a5d9dc4a0fb1336b3bfd777057, State: Running, Role: LEADER
I20250624 07:57:49.183552 19666 consensus_queue.cc:237] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:49.196852 18287 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250624 07:57:49.196421 19185 catalog_manager.cc:5582] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 reported cstate change: term changed from 0 to 1, leader changed from <none> to c076d0a5d9dc4a0fb1336b3bfd777057 (127.17.219.193). New cstate: current_term: 1 leader_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } health_report { overall_health: HEALTHY } } }
I20250624 07:57:49.402192 18287 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250624 07:57:49.692376 19666 consensus_queue.cc:1035] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:57:49.709817 19666 consensus_queue.cc:1035] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:57:49.712491 18287 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 1.1
W20250624 07:57:51.204780 19637 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac140 after lost signal to thread 19513
W20250624 07:57:51.206246 19637 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8940 after lost signal to thread 19640
I20250624 07:57:51.882249 19330 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854"
dest_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36692
I20250624 07:57:51.883003 19330 raft_consensus.cc:604] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 LEADER]: Received request to transfer leadership
I20250624 07:57:52.164796 19706 raft_consensus.cc:991] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057: : Instructing follower 4f470affd74840b2b87fc9a060873aa3 to start an election
I20250624 07:57:52.165226 19677 raft_consensus.cc:1079] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 LEADER]: Signalling peer 4f470affd74840b2b87fc9a060873aa3 to start an election
I20250624 07:57:52.166659 19596 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854"
dest_uuid: "4f470affd74840b2b87fc9a060873aa3"
from {username='slave'} at 127.17.219.193:53383
I20250624 07:57:52.167213 19596 raft_consensus.cc:491] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 07:57:52.167480 19596 raft_consensus.cc:3058] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:57:52.171793 19596 raft_consensus.cc:513] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:52.173923 19596 leader_election.cc:290] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [CANDIDATE]: Term 2 election: Requested vote from peers 1838aabde50e4fbd959daa364f979069 (127.17.219.194:44471), c076d0a5d9dc4a0fb1336b3bfd777057 (127.17.219.193:44873)
I20250624 07:57:52.188258 19330 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854" candidate_uuid: "4f470affd74840b2b87fc9a060873aa3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
I20250624 07:57:52.188619 19463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854" candidate_uuid: "4f470affd74840b2b87fc9a060873aa3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1838aabde50e4fbd959daa364f979069"
I20250624 07:57:52.188935 19330 raft_consensus.cc:3053] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 LEADER]: Stepping down as leader of term 1
I20250624 07:57:52.189076 19463 raft_consensus.cc:3058] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:57:52.189252 19330 raft_consensus.cc:738] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c076d0a5d9dc4a0fb1336b3bfd777057, State: Running, Role: LEADER
I20250624 07:57:52.189925 19330 consensus_queue.cc:260] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:52.191251 19330 raft_consensus.cc:3058] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:57:52.193621 19463 raft_consensus.cc:2466] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f470affd74840b2b87fc9a060873aa3 in term 2.
I20250624 07:57:52.194636 19530 leader_election.cc:304] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1838aabde50e4fbd959daa364f979069, 4f470affd74840b2b87fc9a060873aa3; no voters:
I20250624 07:57:52.196883 19710 raft_consensus.cc:2802] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:57:52.198330 19330 raft_consensus.cc:2466] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f470affd74840b2b87fc9a060873aa3 in term 2.
I20250624 07:57:52.199463 19710 raft_consensus.cc:695] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [term 2 LEADER]: Becoming Leader. State: Replica: 4f470affd74840b2b87fc9a060873aa3, State: Running, Role: LEADER
I20250624 07:57:52.200661 19710 consensus_queue.cc:237] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } }
I20250624 07:57:52.209230 19186 catalog_manager.cc:5582] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 reported cstate change: term changed from 1 to 2, leader changed from c076d0a5d9dc4a0fb1336b3bfd777057 (127.17.219.193) to 4f470affd74840b2b87fc9a060873aa3 (127.17.219.195). New cstate: current_term: 2 leader_uuid: "4f470affd74840b2b87fc9a060873aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f470affd74840b2b87fc9a060873aa3" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 36007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20250624 07:57:52.593459 19463 raft_consensus.cc:1273] T abee3912c65943d2ae21df2e5d2b2854 P 1838aabde50e4fbd959daa364f979069 [term 2 FOLLOWER]: Refusing update from remote peer 4f470affd74840b2b87fc9a060873aa3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 07:57:52.594622 19710 consensus_queue.cc:1035] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1838aabde50e4fbd959daa364f979069" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250624 07:57:52.610904 19330 raft_consensus.cc:1273] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 2 FOLLOWER]: Refusing update from remote peer 4f470affd74840b2b87fc9a060873aa3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 07:57:52.613162 19711 consensus_queue.cc:1035] T abee3912c65943d2ae21df2e5d2b2854 P 4f470affd74840b2b87fc9a060873aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250624 07:57:54.638648 19330 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "abee3912c65943d2ae21df2e5d2b2854"
dest_uuid: "c076d0a5d9dc4a0fb1336b3bfd777057"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36698
I20250624 07:57:54.639271 19330 raft_consensus.cc:604] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 2 FOLLOWER]: Received request to transfer leadership
I20250624 07:57:54.639604 19330 raft_consensus.cc:612] T abee3912c65943d2ae21df2e5d2b2854 P c076d0a5d9dc4a0fb1336b3bfd777057 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250624 07:57:55.679829 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19246
I20250624 07:57:55.703867 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19379
I20250624 07:57:55.728046 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19512
I20250624 07:57:55.755080 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19153
2025-06-24T07:57:55Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14590 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250624 07:57:55.809906 18287 test_util.cc:276] Using random seed: -1348091303
I20250624 07:57:55.813890 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:57:55.814042 18287 ts_itest-base.cc:116] --------------
I20250624 07:57:55.814184 18287 ts_itest-base.cc:117] 3 tablet servers
I20250624 07:57:55.814340 18287 ts_itest-base.cc:118] 3 replicas per TS
I20250624 07:57:55.814471 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:57:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:57:55Z Disabled control of system clock
I20250624 07:57:55.849599 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:37339
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:37339 with env {}
W20250624 07:57:56.145344 19752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:56.145887 19752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:56.146288 19752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:56.177147 19752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:57:56.177429 19752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:56.177639 19752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:57:56.177832 19752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:57:56.212756 19752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:37339
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:37339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:56.213989 19752 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:56.215545 19752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:56.229477 19758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:56.229522 19759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:56.231986 19752 server_base.cc:1048] running on GCE node
W20250624 07:57:56.231844 19761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:57:57.398567 19752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:57.401088 19752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:57.402426 19752 hybrid_clock.cc:648] HybridClock initialized: now 1750751877402387 us; error 50 us; skew 500 ppm
I20250624 07:57:57.403241 19752 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:57.409253 19752 webserver.cc:469] Webserver started at http://127.17.219.254:39285/ using document root <none> and password file <none>
I20250624 07:57:57.410171 19752 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:57.410385 19752 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:57.410866 19752 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:57.416676 19752 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2ebbdf4fcba64770a8c3249ca64bb604"
format_stamp: "Formatted at 2025-06-24 07:57:57 on dist-test-slave-qhgl"
I20250624 07:57:57.417791 19752 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2ebbdf4fcba64770a8c3249ca64bb604"
format_stamp: "Formatted at 2025-06-24 07:57:57 on dist-test-slave-qhgl"
I20250624 07:57:57.425015 19752 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250624 07:57:57.431176 19768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:57.432178 19752 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250624 07:57:57.432494 19752 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ebbdf4fcba64770a8c3249ca64bb604"
format_stamp: "Formatted at 2025-06-24 07:57:57 on dist-test-slave-qhgl"
I20250624 07:57:57.432806 19752 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:57.488793 19752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:57.490247 19752 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:57.490679 19752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:57.559922 19752 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:37339
I20250624 07:57:57.559999 19819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:37339 every 8 connection(s)
I20250624 07:57:57.562547 19752 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:57:57.567687 19820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:57:57.567781 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19752
I20250624 07:57:57.568198 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:57:57.593103 19820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604: Bootstrap starting.
I20250624 07:57:57.598268 19820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604: Neither blocks nor log segments found. Creating new log.
I20250624 07:57:57.599959 19820 log.cc:826] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604: Log is configured to *not* fsync() on all Append() calls
I20250624 07:57:57.604390 19820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604: No bootstrap required, opened a new log
I20250624 07:57:57.621515 19820 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 37339 } }
I20250624 07:57:57.622166 19820 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:57:57.622428 19820 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebbdf4fcba64770a8c3249ca64bb604, State: Initialized, Role: FOLLOWER
I20250624 07:57:57.623152 19820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 37339 } }
I20250624 07:57:57.623637 19820 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:57:57.623977 19820 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:57:57.624310 19820 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:57:57.628502 19820 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 37339 } }
I20250624 07:57:57.629269 19820 leader_election.cc:304] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebbdf4fcba64770a8c3249ca64bb604; no voters:
I20250624 07:57:57.630882 19820 leader_election.cc:290] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:57:57.631610 19825 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:57:57.633730 19825 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebbdf4fcba64770a8c3249ca64bb604, State: Running, Role: LEADER
I20250624 07:57:57.634480 19825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 37339 } }
I20250624 07:57:57.635506 19820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:57:57.644816 19826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 37339 } } }
I20250624 07:57:57.645694 19826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:57.644968 19827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebbdf4fcba64770a8c3249ca64bb604. Latest consensus state: current_term: 1 leader_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebbdf4fcba64770a8c3249ca64bb604" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 37339 } } }
I20250624 07:57:57.646230 19827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604 [sys.catalog]: This master's current role is: LEADER
I20250624 07:57:57.649813 19833 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:57:57.661681 19833 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:57:57.678622 19833 catalog_manager.cc:1349] Generated new cluster ID: 92cf0d79ca48433ab6b24e50956dfadd
I20250624 07:57:57.678952 19833 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:57:57.702694 19833 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:57:57.704703 19833 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:57:57.726346 19833 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2ebbdf4fcba64770a8c3249ca64bb604: Generated new TSK 0
I20250624 07:57:57.727300 19833 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:57:57.744249 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:37339
--builtin_ntp_servers=127.17.219.212:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 07:57:58.044858 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:58.045356 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:58.045850 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:58.076584 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:58.077436 19844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:57:58.111867 19844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:37339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:57:58.113194 19844 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:57:58.114856 19844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:57:58.132014 19850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:58.132380 19851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:58.132071 19853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:57:59.343606 19852 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1210 milliseconds
I20250624 07:57:59.343724 19844 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:57:59.344935 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:57:59.347661 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:57:59.349085 19844 hybrid_clock.cc:648] HybridClock initialized: now 1750751879349025 us; error 76 us; skew 500 ppm
I20250624 07:57:59.349890 19844 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:57:59.357156 19844 webserver.cc:469] Webserver started at http://127.17.219.193:35619/ using document root <none> and password file <none>
I20250624 07:57:59.358091 19844 fs_manager.cc:362] Metadata directory not provided
I20250624 07:57:59.358285 19844 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:57:59.358788 19844 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:57:59.363255 19844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f3055a0efc824d8d9e40175bb872dfc8"
format_stamp: "Formatted at 2025-06-24 07:57:59 on dist-test-slave-qhgl"
I20250624 07:57:59.364337 19844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f3055a0efc824d8d9e40175bb872dfc8"
format_stamp: "Formatted at 2025-06-24 07:57:59 on dist-test-slave-qhgl"
I20250624 07:57:59.372012 19844 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20250624 07:57:59.378017 19860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:59.379132 19844 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:57:59.379448 19844 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f3055a0efc824d8d9e40175bb872dfc8"
format_stamp: "Formatted at 2025-06-24 07:57:59 on dist-test-slave-qhgl"
I20250624 07:57:59.379746 19844 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:57:59.429276 19844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:57:59.430702 19844 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:57:59.431136 19844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:57:59.434090 19844 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:57:59.439224 19844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:57:59.439435 19844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:59.439723 19844 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:57:59.439942 19844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:57:59.609450 19844 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:44347
I20250624 07:57:59.609625 19972 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:44347 every 8 connection(s)
I20250624 07:57:59.612128 19844 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:57:59.613615 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19844
I20250624 07:57:59.614092 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:57:59.622489 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:37339
--builtin_ntp_servers=127.17.219.212:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:57:59.655524 19973 heartbeater.cc:344] Connected to a master server at 127.17.219.254:37339
I20250624 07:57:59.656049 19973 heartbeater.cc:461] Registering TS with master...
I20250624 07:57:59.657406 19973 heartbeater.cc:507] Master 127.17.219.254:37339 requested a full tablet report, sending...
I20250624 07:57:59.660640 19785 ts_manager.cc:194] Registered new tserver with Master: f3055a0efc824d8d9e40175bb872dfc8 (127.17.219.193:44347)
I20250624 07:57:59.663407 19785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:54163
W20250624 07:57:59.937078 19977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:57:59.937598 19977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:57:59.938091 19977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:57:59.969127 19977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:57:59.969990 19977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:58:00.005648 19977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:37339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:00.007001 19977 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:00.008612 19977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:00.027251 19984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:00.667605 19973 heartbeater.cc:499] Master 127.17.219.254:37339 was elected leader, sending a full tablet report...
W20250624 07:58:00.027475 19983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:00.030556 19986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:01.211102 19985 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:58:01.211182 19977 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:01.215165 19977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:01.217285 19977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:01.218644 19977 hybrid_clock.cc:648] HybridClock initialized: now 1750751881218610 us; error 55 us; skew 500 ppm
I20250624 07:58:01.219458 19977 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:01.225873 19977 webserver.cc:469] Webserver started at http://127.17.219.194:45191/ using document root <none> and password file <none>
I20250624 07:58:01.226806 19977 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:01.227035 19977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:01.227500 19977 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:01.231910 19977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc"
format_stamp: "Formatted at 2025-06-24 07:58:01 on dist-test-slave-qhgl"
I20250624 07:58:01.232998 19977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc"
format_stamp: "Formatted at 2025-06-24 07:58:01 on dist-test-slave-qhgl"
I20250624 07:58:01.240016 19977 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250624 07:58:01.245505 19993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:01.246493 19977 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250624 07:58:01.246835 19977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc"
format_stamp: "Formatted at 2025-06-24 07:58:01 on dist-test-slave-qhgl"
I20250624 07:58:01.247190 19977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:01.314285 19977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:01.315770 19977 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:01.316197 19977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:01.318645 19977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:01.322738 19977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:01.322963 19977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250624 07:58:01.323233 19977 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:01.323390 19977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:01.457669 19977 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:40955
I20250624 07:58:01.457769 20105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:40955 every 8 connection(s)
I20250624 07:58:01.460173 19977 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:58:01.466118 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 19977
I20250624 07:58:01.466502 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 07:58:01.472620 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:37339
--builtin_ntp_servers=127.17.219.212:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:01.482273 20106 heartbeater.cc:344] Connected to a master server at 127.17.219.254:37339
I20250624 07:58:01.482896 20106 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:01.484138 20106 heartbeater.cc:507] Master 127.17.219.254:37339 requested a full tablet report, sending...
I20250624 07:58:01.486452 19785 ts_manager.cc:194] Registered new tserver with Master: b6c59a3f112b4967b2cfe5d27abb9fcc (127.17.219.194:40955)
I20250624 07:58:01.488324 19785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:39695
W20250624 07:58:01.774694 20110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:01.775213 20110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:01.775645 20110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:01.806484 20110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:01.807353 20110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:58:01.842038 20110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:37339
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:01.843314 20110 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:01.844889 20110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:01.860249 20116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:02.491838 20106 heartbeater.cc:499] Master 127.17.219.254:37339 was elected leader, sending a full tablet report...
W20250624 07:58:01.860692 20119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:01.862671 20110 server_base.cc:1048] running on GCE node
W20250624 07:58:01.861662 20117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:03.017465 20110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:03.019690 20110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:03.021026 20110 hybrid_clock.cc:648] HybridClock initialized: now 1750751883020991 us; error 54 us; skew 500 ppm
I20250624 07:58:03.021775 20110 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:03.028257 20110 webserver.cc:469] Webserver started at http://127.17.219.195:43467/ using document root <none> and password file <none>
I20250624 07:58:03.029134 20110 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:03.029316 20110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:03.029739 20110 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:03.034067 20110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d220020ca25d4e95a1693219acae1ab0"
format_stamp: "Formatted at 2025-06-24 07:58:03 on dist-test-slave-qhgl"
I20250624 07:58:03.035140 20110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d220020ca25d4e95a1693219acae1ab0"
format_stamp: "Formatted at 2025-06-24 07:58:03 on dist-test-slave-qhgl"
I20250624 07:58:03.042021 20110 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250624 07:58:03.047492 20127 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:03.048473 20110 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250624 07:58:03.048759 20110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d220020ca25d4e95a1693219acae1ab0"
format_stamp: "Formatted at 2025-06-24 07:58:03 on dist-test-slave-qhgl"
I20250624 07:58:03.049053 20110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:03.108554 20110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:03.109963 20110 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:03.110358 20110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:03.112876 20110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:03.116914 20110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:03.117097 20110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:03.117291 20110 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:03.117425 20110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:03.249455 20110 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:44607
I20250624 07:58:03.249574 20239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:44607 every 8 connection(s)
I20250624 07:58:03.252020 20110 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:58:03.253566 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 20110
I20250624 07:58:03.254192 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 07:58:03.272583 20240 heartbeater.cc:344] Connected to a master server at 127.17.219.254:37339
I20250624 07:58:03.273087 20240 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:03.274246 20240 heartbeater.cc:507] Master 127.17.219.254:37339 requested a full tablet report, sending...
I20250624 07:58:03.276399 19785 ts_manager.cc:194] Registered new tserver with Master: d220020ca25d4e95a1693219acae1ab0 (127.17.219.195:44607)
I20250624 07:58:03.277616 19785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:57147
I20250624 07:58:03.289709 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:03.323042 19785 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34944:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 07:58:03.341307 19785 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:58:03.402792 19908 tablet_service.cc:1468] Processing CreateTablet for tablet ae1d69e9ceb44aff83e2677a3f5c6da5 (DEFAULT_TABLE table=TestTable [id=53f40ce9e1c944978866bb04da42a81a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:03.403282 20175 tablet_service.cc:1468] Processing CreateTablet for tablet ae1d69e9ceb44aff83e2677a3f5c6da5 (DEFAULT_TABLE table=TestTable [id=53f40ce9e1c944978866bb04da42a81a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:03.404162 20041 tablet_service.cc:1468] Processing CreateTablet for tablet ae1d69e9ceb44aff83e2677a3f5c6da5 (DEFAULT_TABLE table=TestTable [id=53f40ce9e1c944978866bb04da42a81a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:03.404942 19908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1d69e9ceb44aff83e2677a3f5c6da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:03.405061 20175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1d69e9ceb44aff83e2677a3f5c6da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:03.405956 20041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1d69e9ceb44aff83e2677a3f5c6da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:03.427487 20259 tablet_bootstrap.cc:492] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: Bootstrap starting.
I20250624 07:58:03.433161 20260 tablet_bootstrap.cc:492] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: Bootstrap starting.
I20250624 07:58:03.434517 20259 tablet_bootstrap.cc:654] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:03.438462 20259 log.cc:826] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:03.440855 20260 tablet_bootstrap.cc:654] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:03.442662 20261 tablet_bootstrap.cc:492] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0: Bootstrap starting.
I20250624 07:58:03.443259 20260 log.cc:826] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:03.444542 20259 tablet_bootstrap.cc:492] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: No bootstrap required, opened a new log
I20250624 07:58:03.445071 20259 ts_tablet_manager.cc:1397] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.004s
I20250624 07:58:03.448938 20260 tablet_bootstrap.cc:492] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: No bootstrap required, opened a new log
I20250624 07:58:03.449393 20260 ts_tablet_manager.cc:1397] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250624 07:58:03.450088 20261 tablet_bootstrap.cc:654] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:03.453164 20261 log.cc:826] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:03.472769 20259 raft_consensus.cc:357] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.473704 20259 raft_consensus.cc:383] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:03.474014 20259 raft_consensus.cc:738] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3055a0efc824d8d9e40175bb872dfc8, State: Initialized, Role: FOLLOWER
I20250624 07:58:03.475109 20261 tablet_bootstrap.cc:492] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0: No bootstrap required, opened a new log
I20250624 07:58:03.474936 20259 consensus_queue.cc:260] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.475826 20261 ts_tablet_manager.cc:1397] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.013s
I20250624 07:58:03.481076 20260 raft_consensus.cc:357] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.482237 20260 raft_consensus.cc:383] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:03.482630 20260 raft_consensus.cc:738] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c59a3f112b4967b2cfe5d27abb9fcc, State: Initialized, Role: FOLLOWER
I20250624 07:58:03.483932 20260 consensus_queue.cc:260] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.488159 20259 ts_tablet_manager.cc:1428] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: Time spent starting tablet: real 0.043s user 0.038s sys 0.004s
I20250624 07:58:03.495338 20260 ts_tablet_manager.cc:1428] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: Time spent starting tablet: real 0.046s user 0.040s sys 0.000s
I20250624 07:58:03.507098 20261 raft_consensus.cc:357] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.507764 20261 raft_consensus.cc:383] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:03.508111 20261 raft_consensus.cc:738] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220020ca25d4e95a1693219acae1ab0, State: Initialized, Role: FOLLOWER
I20250624 07:58:03.508808 20261 consensus_queue.cc:260] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.511724 20240 heartbeater.cc:499] Master 127.17.219.254:37339 was elected leader, sending a full tablet report...
I20250624 07:58:03.512009 20261 ts_tablet_manager.cc:1428] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0: Time spent starting tablet: real 0.036s user 0.022s sys 0.007s
I20250624 07:58:03.547935 20265 raft_consensus.cc:491] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:58:03.548410 20265 raft_consensus.cc:513] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.551190 20265 leader_election.cc:290] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220020ca25d4e95a1693219acae1ab0 (127.17.219.195:44607), b6c59a3f112b4967b2cfe5d27abb9fcc (127.17.219.194:40955)
I20250624 07:58:03.562637 20195 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae1d69e9ceb44aff83e2677a3f5c6da5" candidate_uuid: "f3055a0efc824d8d9e40175bb872dfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220020ca25d4e95a1693219acae1ab0" is_pre_election: true
I20250624 07:58:03.563508 20195 raft_consensus.cc:2466] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3055a0efc824d8d9e40175bb872dfc8 in term 0.
I20250624 07:58:03.563746 20061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae1d69e9ceb44aff83e2677a3f5c6da5" candidate_uuid: "f3055a0efc824d8d9e40175bb872dfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" is_pre_election: true
I20250624 07:58:03.564466 20061 raft_consensus.cc:2466] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3055a0efc824d8d9e40175bb872dfc8 in term 0.
I20250624 07:58:03.564823 19863 leader_election.cc:304] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220020ca25d4e95a1693219acae1ab0, f3055a0efc824d8d9e40175bb872dfc8; no voters:
I20250624 07:58:03.565670 20265 raft_consensus.cc:2802] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:58:03.565943 20265 raft_consensus.cc:491] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:58:03.566179 20265 raft_consensus.cc:3058] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:03.570400 20265 raft_consensus.cc:513] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.571763 20265 leader_election.cc:290] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [CANDIDATE]: Term 1 election: Requested vote from peers d220020ca25d4e95a1693219acae1ab0 (127.17.219.195:44607), b6c59a3f112b4967b2cfe5d27abb9fcc (127.17.219.194:40955)
I20250624 07:58:03.572687 20195 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae1d69e9ceb44aff83e2677a3f5c6da5" candidate_uuid: "f3055a0efc824d8d9e40175bb872dfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220020ca25d4e95a1693219acae1ab0"
I20250624 07:58:03.572801 20061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae1d69e9ceb44aff83e2677a3f5c6da5" candidate_uuid: "f3055a0efc824d8d9e40175bb872dfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc"
I20250624 07:58:03.573096 20195 raft_consensus.cc:3058] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:03.573293 20061 raft_consensus.cc:3058] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:03.577648 20195 raft_consensus.cc:2466] T ae1d69e9ceb44aff83e2677a3f5c6da5 P d220020ca25d4e95a1693219acae1ab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3055a0efc824d8d9e40175bb872dfc8 in term 1.
I20250624 07:58:03.578573 19863 leader_election.cc:304] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220020ca25d4e95a1693219acae1ab0, f3055a0efc824d8d9e40175bb872dfc8; no voters:
I20250624 07:58:03.579241 20265 raft_consensus.cc:2802] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:03.579989 20061 raft_consensus.cc:2466] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3055a0efc824d8d9e40175bb872dfc8 in term 1.
I20250624 07:58:03.581926 20265 raft_consensus.cc:695] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [term 1 LEADER]: Becoming Leader. State: Replica: f3055a0efc824d8d9e40175bb872dfc8, State: Running, Role: LEADER
I20250624 07:58:03.582845 20265 consensus_queue.cc:237] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } }
I20250624 07:58:03.595757 19785 catalog_manager.cc:5582] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3055a0efc824d8d9e40175bb872dfc8 (127.17.219.193). New cstate: current_term: 1 leader_uuid: "f3055a0efc824d8d9e40175bb872dfc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } health_report { overall_health: UNKNOWN } } }
W20250624 07:58:03.636222 19974 tablet.cc:2378] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:58:03.644196 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:03.647328 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f3055a0efc824d8d9e40175bb872dfc8 to finish bootstrapping
I20250624 07:58:03.659422 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b6c59a3f112b4967b2cfe5d27abb9fcc to finish bootstrapping
I20250624 07:58:03.670135 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d220020ca25d4e95a1693219acae1ab0 to finish bootstrapping
I20250624 07:58:03.682471 19785 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34944:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250624 07:58:03.683990 19785 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:58:03.699760 19908 tablet_service.cc:1468] Processing CreateTablet for tablet 028620d0784d459597d65e8e6a94504e (DEFAULT_TABLE table=TestAnotherTable [id=232d1aa54a6745829c2bbf01acd94074]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 07:58:03.700474 20041 tablet_service.cc:1468] Processing CreateTablet for tablet 028620d0784d459597d65e8e6a94504e (DEFAULT_TABLE table=TestAnotherTable [id=232d1aa54a6745829c2bbf01acd94074]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 07:58:03.700846 19908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 028620d0784d459597d65e8e6a94504e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:03.700906 20175 tablet_service.cc:1468] Processing CreateTablet for tablet 028620d0784d459597d65e8e6a94504e (DEFAULT_TABLE table=TestAnotherTable [id=232d1aa54a6745829c2bbf01acd94074]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 07:58:03.701292 20041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 028620d0784d459597d65e8e6a94504e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:03.701923 20175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 028620d0784d459597d65e8e6a94504e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:03.711225 20260 tablet_bootstrap.cc:492] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc: Bootstrap starting.
I20250624 07:58:03.714974 20259 tablet_bootstrap.cc:492] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8: Bootstrap starting.
I20250624 07:58:03.715770 20261 tablet_bootstrap.cc:492] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0: Bootstrap starting.
I20250624 07:58:03.717479 20260 tablet_bootstrap.cc:654] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc: Neither blocks nor log segments found. Creating new log.
W20250624 07:58:03.719709 20107 tablet.cc:2378] T ae1d69e9ceb44aff83e2677a3f5c6da5 P b6c59a3f112b4967b2cfe5d27abb9fcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:58:03.721060 20259 tablet_bootstrap.cc:654] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:03.721506 20261 tablet_bootstrap.cc:654] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:03.723394 20260 tablet_bootstrap.cc:492] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc: No bootstrap required, opened a new log
I20250624 07:58:03.723907 20260 ts_tablet_manager.cc:1397] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20250624 07:58:03.727038 20260 raft_consensus.cc:357] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.727670 20260 raft_consensus.cc:383] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:03.727893 20260 raft_consensus.cc:738] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c59a3f112b4967b2cfe5d27abb9fcc, State: Initialized, Role: FOLLOWER
I20250624 07:58:03.728431 20259 tablet_bootstrap.cc:492] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8: No bootstrap required, opened a new log
I20250624 07:58:03.728694 20261 tablet_bootstrap.cc:492] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0: No bootstrap required, opened a new log
I20250624 07:58:03.728435 20260 consensus_queue.cc:260] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.728828 20259 ts_tablet_manager.cc:1397] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250624 07:58:03.729061 20261 ts_tablet_manager.cc:1397] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20250624 07:58:03.730255 20260 ts_tablet_manager.cc:1428] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20250624 07:58:03.731400 20259 raft_consensus.cc:357] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.732075 20259 raft_consensus.cc:383] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:03.731971 20261 raft_consensus.cc:357] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.732323 20259 raft_consensus.cc:738] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3055a0efc824d8d9e40175bb872dfc8, State: Initialized, Role: FOLLOWER
I20250624 07:58:03.732641 20261 raft_consensus.cc:383] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:03.732904 20261 raft_consensus.cc:738] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220020ca25d4e95a1693219acae1ab0, State: Initialized, Role: FOLLOWER
I20250624 07:58:03.732932 20259 consensus_queue.cc:260] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.733551 20261 consensus_queue.cc:260] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.735596 20261 ts_tablet_manager.cc:1428] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20250624 07:58:03.736901 20259 ts_tablet_manager.cc:1428] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
W20250624 07:58:03.757200 20241 tablet.cc:2378] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:58:03.815161 20266 raft_consensus.cc:491] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:58:03.815557 20266 raft_consensus.cc:513] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.817976 20266 leader_election.cc:290] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3055a0efc824d8d9e40175bb872dfc8 (127.17.219.193:44347), d220020ca25d4e95a1693219acae1ab0 (127.17.219.195:44607)
I20250624 07:58:03.828917 20195 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "028620d0784d459597d65e8e6a94504e" candidate_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220020ca25d4e95a1693219acae1ab0" is_pre_election: true
I20250624 07:58:03.829147 19928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "028620d0784d459597d65e8e6a94504e" candidate_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3055a0efc824d8d9e40175bb872dfc8" is_pre_election: true
I20250624 07:58:03.829579 20195 raft_consensus.cc:2466] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c59a3f112b4967b2cfe5d27abb9fcc in term 0.
I20250624 07:58:03.829888 19928 raft_consensus.cc:2466] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c59a3f112b4967b2cfe5d27abb9fcc in term 0.
I20250624 07:58:03.830704 19996 leader_election.cc:304] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c59a3f112b4967b2cfe5d27abb9fcc, d220020ca25d4e95a1693219acae1ab0; no voters:
I20250624 07:58:03.831552 20266 raft_consensus.cc:2802] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:58:03.831807 20266 raft_consensus.cc:491] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:58:03.832053 20266 raft_consensus.cc:3058] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:03.836638 20266 raft_consensus.cc:513] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.838114 20266 leader_election.cc:290] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [CANDIDATE]: Term 1 election: Requested vote from peers f3055a0efc824d8d9e40175bb872dfc8 (127.17.219.193:44347), d220020ca25d4e95a1693219acae1ab0 (127.17.219.195:44607)
I20250624 07:58:03.838866 19928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "028620d0784d459597d65e8e6a94504e" candidate_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3055a0efc824d8d9e40175bb872dfc8"
I20250624 07:58:03.838997 20195 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "028620d0784d459597d65e8e6a94504e" candidate_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220020ca25d4e95a1693219acae1ab0"
I20250624 07:58:03.839382 19928 raft_consensus.cc:3058] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:03.839401 20195 raft_consensus.cc:3058] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:03.843360 20195 raft_consensus.cc:2466] T 028620d0784d459597d65e8e6a94504e P d220020ca25d4e95a1693219acae1ab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c59a3f112b4967b2cfe5d27abb9fcc in term 1.
I20250624 07:58:03.843617 19928 raft_consensus.cc:2466] T 028620d0784d459597d65e8e6a94504e P f3055a0efc824d8d9e40175bb872dfc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c59a3f112b4967b2cfe5d27abb9fcc in term 1.
I20250624 07:58:03.844146 19996 leader_election.cc:304] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c59a3f112b4967b2cfe5d27abb9fcc, d220020ca25d4e95a1693219acae1ab0; no voters:
I20250624 07:58:03.844710 20266 raft_consensus.cc:2802] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:03.846156 20266 raft_consensus.cc:695] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [term 1 LEADER]: Becoming Leader. State: Replica: b6c59a3f112b4967b2cfe5d27abb9fcc, State: Running, Role: LEADER
I20250624 07:58:03.847002 20266 consensus_queue.cc:237] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } }
I20250624 07:58:03.858035 19783 catalog_manager.cc:5582] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c59a3f112b4967b2cfe5d27abb9fcc (127.17.219.194). New cstate: current_term: 1 leader_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 } health_report { overall_health: UNKNOWN } } }
I20250624 07:58:04.066555 20265 consensus_queue.cc:1035] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:58:04.096333 20282 consensus_queue.cc:1035] T ae1d69e9ceb44aff83e2677a3f5c6da5 P f3055a0efc824d8d9e40175bb872dfc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6c59a3f112b4967b2cfe5d27abb9fcc" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250624 07:58:04.187320 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:04.187918 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:04.219513 20280 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 07:58:04.287102 20266 consensus_queue.cc:1035] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3055a0efc824d8d9e40175bb872dfc8" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 07:58:04.314800 20266 consensus_queue.cc:1035] T 028620d0784d459597d65e8e6a94504e P b6c59a3f112b4967b2cfe5d27abb9fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220020ca25d4e95a1693219acae1ab0" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250624 07:58:05.578564 20280 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.311s user 0.483s sys 0.811s
W20250624 07:58:05.578997 20280 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.311s user 0.486s sys 0.811s
W20250624 07:58:06.953094 20305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:06.953689 20305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:06.985460 20305 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 07:58:08.318910 20305 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.292s user 0.458s sys 0.833s
W20250624 07:58:08.319308 20305 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.292s user 0.458s sys 0.833s
W20250624 07:58:09.689767 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:09.690311 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:09.721040 20322 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 07:58:11.043550 20322 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.282s user 0.430s sys 0.851s
W20250624 07:58:11.043948 20322 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.282s user 0.430s sys 0.851s
W20250624 07:58:11.150058 20102 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd0e0 after lost signal to thread 19978
W20250624 07:58:11.150951 20102 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd0a0 after lost signal to thread 20105
W20250624 07:58:12.497655 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:12.498262 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:12.530369 20338 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 07:58:13.893999 20338 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.316s user 0.000s sys 0.002s
W20250624 07:58:13.894282 20338 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.316s user 0.000s sys 0.002s
I20250624 07:58:14.985706 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19844
I20250624 07:58:15.016024 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19977
I20250624 07:58:15.046263 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 20110
I20250624 07:58:15.071336 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 19752
2025-06-24T07:58:15Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19325 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250624 07:58:15.136461 18287 test_util.cc:276] Using random seed: -1328764757
I20250624 07:58:15.141158 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:58:15.141362 18287 ts_itest-base.cc:116] --------------
I20250624 07:58:15.141525 18287 ts_itest-base.cc:117] 3 tablet servers
I20250624 07:58:15.141674 18287 ts_itest-base.cc:118] 3 replicas per TS
I20250624 07:58:15.141829 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:58:15Z Disabled control of system clock
I20250624 07:58:15.187367 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:38521
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:38521
--superuser_acl=no-such-user with env {}
W20250624 07:58:15.517524 20360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:15.518126 20360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:15.518580 20360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:15.550340 20360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:58:15.550668 20360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:15.550951 20360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:58:15.551185 20360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:58:15.586843 20360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:38521
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:15.588279 20360 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:15.590126 20360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:15.604918 20366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:15.605579 20369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:15.605880 20367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:15.607244 20360 server_base.cc:1048] running on GCE node
I20250624 07:58:16.790102 20360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:16.793246 20360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:16.794677 20360 hybrid_clock.cc:648] HybridClock initialized: now 1750751896794636 us; error 57 us; skew 500 ppm
I20250624 07:58:16.795508 20360 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:16.806449 20360 webserver.cc:469] Webserver started at http://127.17.219.254:33103/ using document root <none> and password file <none>
I20250624 07:58:16.807433 20360 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:16.807637 20360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:16.808105 20360 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:16.812681 20360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3cf6995bb59149849a624f8d6345efed"
format_stamp: "Formatted at 2025-06-24 07:58:16 on dist-test-slave-qhgl"
I20250624 07:58:16.813802 20360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3cf6995bb59149849a624f8d6345efed"
format_stamp: "Formatted at 2025-06-24 07:58:16 on dist-test-slave-qhgl"
I20250624 07:58:16.821010 20360 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250624 07:58:16.826367 20376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:16.827364 20360 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250624 07:58:16.827675 20360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3cf6995bb59149849a624f8d6345efed"
format_stamp: "Formatted at 2025-06-24 07:58:16 on dist-test-slave-qhgl"
I20250624 07:58:16.828001 20360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:16.877084 20360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:16.878511 20360 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:16.878952 20360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:16.948961 20360 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:38521
I20250624 07:58:16.949030 20427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:38521 every 8 connection(s)
I20250624 07:58:16.951699 20360 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:58:16.952414 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 20360
I20250624 07:58:16.952804 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:58:16.958616 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:16.980262 20428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed: Bootstrap starting.
I20250624 07:58:16.985700 20428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:16.987972 20428 log.cc:826] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:16.992197 20428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed: No bootstrap required, opened a new log
I20250624 07:58:17.010517 20428 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf6995bb59149849a624f8d6345efed" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 38521 } }
I20250624 07:58:17.011457 20428 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:17.011762 20428 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf6995bb59149849a624f8d6345efed, State: Initialized, Role: FOLLOWER
I20250624 07:58:17.012629 20428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf6995bb59149849a624f8d6345efed" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 38521 } }
I20250624 07:58:17.013293 20428 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:17.013540 20428 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:17.013810 20428 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:17.017951 20428 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf6995bb59149849a624f8d6345efed" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 38521 } }
I20250624 07:58:17.018679 20428 leader_election.cc:304] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cf6995bb59149849a624f8d6345efed; no voters:
I20250624 07:58:17.020339 20428 leader_election.cc:290] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:58:17.021055 20433 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:17.023195 20433 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [term 1 LEADER]: Becoming Leader. State: Replica: 3cf6995bb59149849a624f8d6345efed, State: Running, Role: LEADER
I20250624 07:58:17.024099 20433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf6995bb59149849a624f8d6345efed" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 38521 } }
I20250624 07:58:17.024493 20428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:58:17.035184 20434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cf6995bb59149849a624f8d6345efed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf6995bb59149849a624f8d6345efed" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 38521 } } }
I20250624 07:58:17.035070 20435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf6995bb59149849a624f8d6345efed. Latest consensus state: current_term: 1 leader_uuid: "3cf6995bb59149849a624f8d6345efed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf6995bb59149849a624f8d6345efed" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 38521 } } }
I20250624 07:58:17.035815 20434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:17.035950 20435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:17.040167 20442 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:58:17.052469 20442 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:58:17.067701 20442 catalog_manager.cc:1349] Generated new cluster ID: ff0e2cbd1be347d39743515fcaacf191
I20250624 07:58:17.068023 20442 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:58:17.084035 20442 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:58:17.085533 20442 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:58:17.100935 20442 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3cf6995bb59149849a624f8d6345efed: Generated new TSK 0
I20250624 07:58:17.101872 20442 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:58:17.116600 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:38521
--builtin_ntp_servers=127.17.219.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 07:58:17.424571 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:17.425076 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:17.425618 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:17.456835 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:17.457695 20452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:58:17.492343 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:38521
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:17.493698 20452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:17.495406 20452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:17.513845 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:17.522948 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:18.951607 20460 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1439 milliseconds
I20250624 07:58:18.952091 20452 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250624 07:58:17.514180 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:18.957265 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:18.960017 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:18.961531 20452 hybrid_clock.cc:648] HybridClock initialized: now 1750751898961481 us; error 52 us; skew 500 ppm
I20250624 07:58:18.962621 20452 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:18.970281 20452 webserver.cc:469] Webserver started at http://127.17.219.193:34731/ using document root <none> and password file <none>
I20250624 07:58:18.971552 20452 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:18.971829 20452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:18.972421 20452 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:18.979010 20452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8d249d96e86e4fbdab2d1e893f1f4929"
format_stamp: "Formatted at 2025-06-24 07:58:18 on dist-test-slave-qhgl"
I20250624 07:58:18.980593 20452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8d249d96e86e4fbdab2d1e893f1f4929"
format_stamp: "Formatted at 2025-06-24 07:58:18 on dist-test-slave-qhgl"
I20250624 07:58:18.990175 20452 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250624 07:58:18.997713 20468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:18.998926 20452 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:58:18.999321 20452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8d249d96e86e4fbdab2d1e893f1f4929"
format_stamp: "Formatted at 2025-06-24 07:58:18 on dist-test-slave-qhgl"
I20250624 07:58:18.999763 20452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:19.077841 20452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:19.079805 20452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:19.080410 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:19.083664 20452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:19.089303 20452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:19.089583 20452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:19.089881 20452 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:19.090111 20452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:19.241878 20452 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:44881
I20250624 07:58:19.241974 20580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:44881 every 8 connection(s)
I20250624 07:58:19.245486 20452 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:58:19.248354 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 20452
I20250624 07:58:19.248842 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:58:19.260834 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:38521
--builtin_ntp_servers=127.17.219.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:19.286867 20581 heartbeater.cc:344] Connected to a master server at 127.17.219.254:38521
I20250624 07:58:19.287410 20581 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:19.288671 20581 heartbeater.cc:507] Master 127.17.219.254:38521 requested a full tablet report, sending...
I20250624 07:58:19.291715 20393 ts_manager.cc:194] Registered new tserver with Master: 8d249d96e86e4fbdab2d1e893f1f4929 (127.17.219.193:44881)
I20250624 07:58:19.294389 20393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:56117
W20250624 07:58:19.572265 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:19.572794 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:19.573309 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:19.604482 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:19.605367 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:58:19.639748 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:38521
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:19.641089 20585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:19.642743 20585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:19.659883 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:20.298542 20581 heartbeater.cc:499] Master 127.17.219.254:38521 was elected leader, sending a full tablet report...
W20250624 07:58:19.661907 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:19.660298 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:20.857615 20593 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1194 milliseconds
I20250624 07:58:20.857710 20585 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:20.859041 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:20.861603 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:20.863034 20585 hybrid_clock.cc:648] HybridClock initialized: now 1750751900862969 us; error 61 us; skew 500 ppm
I20250624 07:58:20.864421 20585 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:20.872327 20585 webserver.cc:469] Webserver started at http://127.17.219.194:43997/ using document root <none> and password file <none>
I20250624 07:58:20.873625 20585 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:20.873828 20585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:20.874419 20585 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:20.881014 20585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "274b4f4ab1be4bac9d979f5782d2df60"
format_stamp: "Formatted at 2025-06-24 07:58:20 on dist-test-slave-qhgl"
I20250624 07:58:20.882122 20585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "274b4f4ab1be4bac9d979f5782d2df60"
format_stamp: "Formatted at 2025-06-24 07:58:20 on dist-test-slave-qhgl"
I20250624 07:58:20.890303 20585 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20250624 07:58:20.896936 20601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:20.897929 20585 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250624 07:58:20.898232 20585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "274b4f4ab1be4bac9d979f5782d2df60"
format_stamp: "Formatted at 2025-06-24 07:58:20 on dist-test-slave-qhgl"
I20250624 07:58:20.898578 20585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:20.965648 20585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:20.967171 20585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:20.967571 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:20.970014 20585 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:20.974068 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:20.974277 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:20.974470 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:20.974615 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:21.107549 20585 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:40301
I20250624 07:58:21.107657 20713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:40301 every 8 connection(s)
I20250624 07:58:21.110040 20585 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:58:21.116602 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 20585
I20250624 07:58:21.117146 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 07:58:21.124392 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:38521
--builtin_ntp_servers=127.17.219.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:21.133533 20714 heartbeater.cc:344] Connected to a master server at 127.17.219.254:38521
I20250624 07:58:21.133952 20714 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:21.134977 20714 heartbeater.cc:507] Master 127.17.219.254:38521 requested a full tablet report, sending...
I20250624 07:58:21.136967 20393 ts_manager.cc:194] Registered new tserver with Master: 274b4f4ab1be4bac9d979f5782d2df60 (127.17.219.194:40301)
I20250624 07:58:21.138254 20393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:45573
W20250624 07:58:21.426522 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:21.427049 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:21.427541 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:21.458479 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:21.459367 20718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:58:21.493985 20718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:38521
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:21.495293 20718 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:21.496886 20718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:21.512362 20725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:22.141281 20714 heartbeater.cc:499] Master 127.17.219.254:38521 was elected leader, sending a full tablet report...
W20250624 07:58:21.512353 20724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:21.513172 20727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:21.514647 20718 server_base.cc:1048] running on GCE node
I20250624 07:58:22.691779 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:22.693959 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:22.695292 20718 hybrid_clock.cc:648] HybridClock initialized: now 1750751902695262 us; error 40 us; skew 500 ppm
I20250624 07:58:22.696149 20718 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:22.705497 20718 webserver.cc:469] Webserver started at http://127.17.219.195:35749/ using document root <none> and password file <none>
I20250624 07:58:22.706423 20718 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:22.706641 20718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:22.707130 20718 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:22.711560 20718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "816c5b23ba394723aa2ef8bf8789a7e6"
format_stamp: "Formatted at 2025-06-24 07:58:22 on dist-test-slave-qhgl"
I20250624 07:58:22.712643 20718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "816c5b23ba394723aa2ef8bf8789a7e6"
format_stamp: "Formatted at 2025-06-24 07:58:22 on dist-test-slave-qhgl"
I20250624 07:58:22.719626 20718 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250624 07:58:22.725025 20735 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:22.726014 20718 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250624 07:58:22.726322 20718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "816c5b23ba394723aa2ef8bf8789a7e6"
format_stamp: "Formatted at 2025-06-24 07:58:22 on dist-test-slave-qhgl"
I20250624 07:58:22.726657 20718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:22.774483 20718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:22.775970 20718 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:22.776381 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:22.778796 20718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:22.782701 20718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:22.782932 20718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250624 07:58:22.783170 20718 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:22.783339 20718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:22.915872 20718 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:37383
I20250624 07:58:22.915968 20847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:37383 every 8 connection(s)
I20250624 07:58:22.918507 20718 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:58:22.928025 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 20718
I20250624 07:58:22.928525 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 07:58:22.938956 20848 heartbeater.cc:344] Connected to a master server at 127.17.219.254:38521
I20250624 07:58:22.939348 20848 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:22.940284 20848 heartbeater.cc:507] Master 127.17.219.254:38521 requested a full tablet report, sending...
I20250624 07:58:22.942322 20393 ts_manager.cc:194] Registered new tserver with Master: 816c5b23ba394723aa2ef8bf8789a7e6 (127.17.219.195:37383)
I20250624 07:58:22.943575 20393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:56237
I20250624 07:58:22.948374 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:22.982137 20393 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33198:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 07:58:23.000487 20393 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:58:23.065557 20783 tablet_service.cc:1468] Processing CreateTablet for tablet 885d9c4849aa418caca35e036af3f5a1 (DEFAULT_TABLE table=TestTable [id=22db6766de6d4e8fad7a97b077fdeb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:23.066143 20649 tablet_service.cc:1468] Processing CreateTablet for tablet 885d9c4849aa418caca35e036af3f5a1 (DEFAULT_TABLE table=TestTable [id=22db6766de6d4e8fad7a97b077fdeb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:23.067287 20783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885d9c4849aa418caca35e036af3f5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:23.067862 20649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885d9c4849aa418caca35e036af3f5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:23.075153 20516 tablet_service.cc:1468] Processing CreateTablet for tablet 885d9c4849aa418caca35e036af3f5a1 (DEFAULT_TABLE table=TestTable [id=22db6766de6d4e8fad7a97b077fdeb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:23.076941 20516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885d9c4849aa418caca35e036af3f5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:23.089192 20867 tablet_bootstrap.cc:492] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: Bootstrap starting.
I20250624 07:58:23.094496 20867 tablet_bootstrap.cc:654] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:23.096913 20867 log.cc:826] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:23.099052 20868 tablet_bootstrap.cc:492] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: Bootstrap starting.
I20250624 07:58:23.103364 20867 tablet_bootstrap.cc:492] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: No bootstrap required, opened a new log
I20250624 07:58:23.104048 20867 ts_tablet_manager.cc:1397] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20250624 07:58:23.105226 20870 tablet_bootstrap.cc:492] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: Bootstrap starting.
I20250624 07:58:23.106396 20868 tablet_bootstrap.cc:654] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:23.108857 20868 log.cc:826] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:23.112774 20870 tablet_bootstrap.cc:654] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:23.115191 20870 log.cc:826] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:23.126457 20870 tablet_bootstrap.cc:492] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: No bootstrap required, opened a new log
I20250624 07:58:23.127102 20870 ts_tablet_manager.cc:1397] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.004s
I20250624 07:58:23.129308 20868 tablet_bootstrap.cc:492] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: No bootstrap required, opened a new log
I20250624 07:58:23.129882 20868 ts_tablet_manager.cc:1397] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.019s
I20250624 07:58:23.135268 20867 raft_consensus.cc:357] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.136298 20867 raft_consensus.cc:383] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:23.136631 20867 raft_consensus.cc:738] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816c5b23ba394723aa2ef8bf8789a7e6, State: Initialized, Role: FOLLOWER
I20250624 07:58:23.137728 20867 consensus_queue.cc:260] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.153802 20870 raft_consensus.cc:357] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.154830 20870 raft_consensus.cc:383] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:23.155194 20870 raft_consensus.cc:738] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d249d96e86e4fbdab2d1e893f1f4929, State: Initialized, Role: FOLLOWER
I20250624 07:58:23.156052 20848 heartbeater.cc:499] Master 127.17.219.254:38521 was elected leader, sending a full tablet report...
I20250624 07:58:23.156224 20870 consensus_queue.cc:260] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.157465 20867 ts_tablet_manager.cc:1428] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: Time spent starting tablet: real 0.053s user 0.028s sys 0.018s
I20250624 07:58:23.163122 20868 raft_consensus.cc:357] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.164296 20868 raft_consensus.cc:383] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:23.166507 20870 ts_tablet_manager.cc:1428] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: Time spent starting tablet: real 0.039s user 0.032s sys 0.004s
I20250624 07:58:23.164641 20868 raft_consensus.cc:738] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274b4f4ab1be4bac9d979f5782d2df60, State: Initialized, Role: FOLLOWER
I20250624 07:58:23.168046 20868 consensus_queue.cc:260] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.172070 20868 ts_tablet_manager.cc:1428] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: Time spent starting tablet: real 0.042s user 0.032s sys 0.001s
W20250624 07:58:23.173020 20849 tablet.cc:2378] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:58:23.270694 20582 tablet.cc:2378] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:58:23.366297 20715 tablet.cc:2378] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:58:23.463019 20874 raft_consensus.cc:491] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:58:23.463447 20874 raft_consensus.cc:513] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.465785 20874 leader_election.cc:290] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274b4f4ab1be4bac9d979f5782d2df60 (127.17.219.194:40301), 816c5b23ba394723aa2ef8bf8789a7e6 (127.17.219.195:37383)
I20250624 07:58:23.476902 20669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885d9c4849aa418caca35e036af3f5a1" candidate_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274b4f4ab1be4bac9d979f5782d2df60" is_pre_election: true
I20250624 07:58:23.477619 20803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885d9c4849aa418caca35e036af3f5a1" candidate_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" is_pre_election: true
I20250624 07:58:23.477835 20669 raft_consensus.cc:2466] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d249d96e86e4fbdab2d1e893f1f4929 in term 0.
I20250624 07:58:23.478381 20803 raft_consensus.cc:2466] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d249d96e86e4fbdab2d1e893f1f4929 in term 0.
I20250624 07:58:23.479118 20470 leader_election.cc:304] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274b4f4ab1be4bac9d979f5782d2df60, 8d249d96e86e4fbdab2d1e893f1f4929; no voters:
I20250624 07:58:23.479784 20874 raft_consensus.cc:2802] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:58:23.480038 20874 raft_consensus.cc:491] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:58:23.480269 20874 raft_consensus.cc:3058] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:23.484561 20874 raft_consensus.cc:513] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.485884 20874 leader_election.cc:290] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [CANDIDATE]: Term 1 election: Requested vote from peers 274b4f4ab1be4bac9d979f5782d2df60 (127.17.219.194:40301), 816c5b23ba394723aa2ef8bf8789a7e6 (127.17.219.195:37383)
I20250624 07:58:23.486646 20669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885d9c4849aa418caca35e036af3f5a1" candidate_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274b4f4ab1be4bac9d979f5782d2df60"
I20250624 07:58:23.486838 20803 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885d9c4849aa418caca35e036af3f5a1" candidate_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816c5b23ba394723aa2ef8bf8789a7e6"
I20250624 07:58:23.487219 20669 raft_consensus.cc:3058] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:23.487345 20803 raft_consensus.cc:3058] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:23.494026 20669 raft_consensus.cc:2466] T 885d9c4849aa418caca35e036af3f5a1 P 274b4f4ab1be4bac9d979f5782d2df60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d249d96e86e4fbdab2d1e893f1f4929 in term 1.
I20250624 07:58:23.494026 20803 raft_consensus.cc:2466] T 885d9c4849aa418caca35e036af3f5a1 P 816c5b23ba394723aa2ef8bf8789a7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d249d96e86e4fbdab2d1e893f1f4929 in term 1.
I20250624 07:58:23.495110 20469 leader_election.cc:304] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816c5b23ba394723aa2ef8bf8789a7e6, 8d249d96e86e4fbdab2d1e893f1f4929; no voters:
I20250624 07:58:23.495734 20874 raft_consensus.cc:2802] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:23.497335 20874 raft_consensus.cc:695] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [term 1 LEADER]: Becoming Leader. State: Replica: 8d249d96e86e4fbdab2d1e893f1f4929, State: Running, Role: LEADER
I20250624 07:58:23.498159 20874 consensus_queue.cc:237] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } }
I20250624 07:58:23.508761 20392 catalog_manager.cc:5582] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d249d96e86e4fbdab2d1e893f1f4929 (127.17.219.193). New cstate: current_term: 1 leader_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d249d96e86e4fbdab2d1e893f1f4929" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 44881 } health_report { overall_health: HEALTHY } } }
I20250624 07:58:23.628340 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:23.631531 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8d249d96e86e4fbdab2d1e893f1f4929 to finish bootstrapping
I20250624 07:58:23.643471 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 274b4f4ab1be4bac9d979f5782d2df60 to finish bootstrapping
I20250624 07:58:23.653213 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 816c5b23ba394723aa2ef8bf8789a7e6 to finish bootstrapping
I20250624 07:58:23.948187 20874 consensus_queue.cc:1035] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816c5b23ba394723aa2ef8bf8789a7e6" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:58:24.007719 20878 consensus_queue.cc:1035] T 885d9c4849aa418caca35e036af3f5a1 P 8d249d96e86e4fbdab2d1e893f1f4929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274b4f4ab1be4bac9d979f5782d2df60" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 07:58:25.348733 20392 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:33208
I20250624 07:58:26.379918 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 20452
I20250624 07:58:26.406905 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 20585
I20250624 07:58:26.433836 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 20718
I20250624 07:58:26.459136 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 20360
2025-06-24T07:58:26Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11376 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250624 07:58:26.512823 18287 test_util.cc:276] Using random seed: -1317388395
I20250624 07:58:26.517414 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:58:26.517594 18287 ts_itest-base.cc:116] --------------
I20250624 07:58:26.517760 18287 ts_itest-base.cc:117] 3 tablet servers
I20250624 07:58:26.517901 18287 ts_itest-base.cc:118] 3 replicas per TS
I20250624 07:58:26.518046 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:58:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:58:26Z Disabled control of system clock
I20250624 07:58:26.563827 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:35921
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:35921 with env {}
W20250624 07:58:26.858494 20916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:26.859133 20916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:26.859588 20916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:26.890269 20916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:58:26.890615 20916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:26.890894 20916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:58:26.891144 20916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:58:26.926106 20916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:35921
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:26.927476 20916 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:26.929093 20916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:26.944945 20922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:26.944984 20925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:26.947606 20916 server_base.cc:1048] running on GCE node
W20250624 07:58:26.945997 20923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:28.124105 20916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:28.127202 20916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:28.128628 20916 hybrid_clock.cc:648] HybridClock initialized: now 1750751908128575 us; error 70 us; skew 500 ppm
I20250624 07:58:28.129457 20916 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:28.140936 20916 webserver.cc:469] Webserver started at http://127.17.219.254:45265/ using document root <none> and password file <none>
I20250624 07:58:28.141959 20916 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:28.142141 20916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:28.142558 20916 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:28.148099 20916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "58dfc8c4b9e0494280f8f9e271bd4459"
format_stamp: "Formatted at 2025-06-24 07:58:28 on dist-test-slave-qhgl"
I20250624 07:58:28.149194 20916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "58dfc8c4b9e0494280f8f9e271bd4459"
format_stamp: "Formatted at 2025-06-24 07:58:28 on dist-test-slave-qhgl"
I20250624 07:58:28.156163 20916 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250624 07:58:28.161545 20932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:28.162494 20916 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250624 07:58:28.162817 20916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "58dfc8c4b9e0494280f8f9e271bd4459"
format_stamp: "Formatted at 2025-06-24 07:58:28 on dist-test-slave-qhgl"
I20250624 07:58:28.163122 20916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:28.212450 20916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:28.213878 20916 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:28.214305 20916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:28.283185 20916 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:35921
I20250624 07:58:28.283267 20983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:35921 every 8 connection(s)
I20250624 07:58:28.285812 20916 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:58:28.290835 20984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:28.296654 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 20916
I20250624 07:58:28.297225 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:58:28.311081 20984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap starting.
I20250624 07:58:28.316689 20984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:28.318522 20984 log.cc:826] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:28.323794 20984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: No bootstrap required, opened a new log
I20250624 07:58:28.342375 20984 raft_consensus.cc:357] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:28.343122 20984 raft_consensus.cc:383] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:28.343369 20984 raft_consensus.cc:738] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Initialized, Role: FOLLOWER
I20250624 07:58:28.344034 20984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:28.344549 20984 raft_consensus.cc:397] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:28.344803 20984 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:28.345108 20984 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:28.349184 20984 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:28.349905 20984 leader_election.cc:304] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dfc8c4b9e0494280f8f9e271bd4459; no voters:
I20250624 07:58:28.351647 20984 leader_election.cc:290] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:58:28.352569 20989 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:28.355053 20989 raft_consensus.cc:695] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 1 LEADER]: Becoming Leader. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Running, Role: LEADER
I20250624 07:58:28.355551 20984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:58:28.355835 20989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:28.367254 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dfc8c4b9e0494280f8f9e271bd4459. Latest consensus state: current_term: 1 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:28.367472 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:28.368155 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:28.368155 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:28.371803 20997 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:58:28.382620 20997 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:58:28.400060 20997 catalog_manager.cc:1349] Generated new cluster ID: f025185adc7b4d40a894b43d1ebfec9b
I20250624 07:58:28.400444 20997 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:58:28.431654 20997 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:58:28.433133 20997 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:58:28.445971 20997 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Generated new TSK 0
I20250624 07:58:28.446856 20997 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:58:28.467674 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:35921
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 07:58:28.786389 21008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:28.786902 21008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:28.787417 21008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:28.818650 21008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:28.819505 21008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:58:28.854795 21008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:28.856091 21008 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:28.857801 21008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:28.874683 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:30.274672 21013 debug-util.cc:398] Leaking SignalData structure 0x7b08000184e0 after lost signal to thread 21008
W20250624 07:58:30.734187 21008 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.863s user 0.661s sys 1.124s
W20250624 07:58:28.875496 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:30.734599 21008 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.863s user 0.661s sys 1.124s
W20250624 07:58:30.736464 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:30.740163 21016 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1863 milliseconds
I20250624 07:58:30.740172 21008 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:30.741498 21008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:30.743649 21008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:30.744992 21008 hybrid_clock.cc:648] HybridClock initialized: now 1750751910744947 us; error 55 us; skew 500 ppm
I20250624 07:58:30.745770 21008 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:30.753079 21008 webserver.cc:469] Webserver started at http://127.17.219.193:46399/ using document root <none> and password file <none>
I20250624 07:58:30.753974 21008 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:30.754191 21008 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:30.754627 21008 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:30.759485 21008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "73f2526ee2e54386be70659b58fe5318"
format_stamp: "Formatted at 2025-06-24 07:58:30 on dist-test-slave-qhgl"
I20250624 07:58:30.760535 21008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "73f2526ee2e54386be70659b58fe5318"
format_stamp: "Formatted at 2025-06-24 07:58:30 on dist-test-slave-qhgl"
I20250624 07:58:30.767652 21008 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250624 07:58:30.773820 21024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:30.774952 21008 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:58:30.775272 21008 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73f2526ee2e54386be70659b58fe5318"
format_stamp: "Formatted at 2025-06-24 07:58:30 on dist-test-slave-qhgl"
I20250624 07:58:30.775674 21008 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:30.832381 21008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:30.833806 21008 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:30.834244 21008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:30.837271 21008 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:30.841641 21008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:30.841849 21008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:30.842090 21008 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:30.842245 21008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:31.029688 21008 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:41225
I20250624 07:58:31.029776 21136 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:41225 every 8 connection(s)
I20250624 07:58:31.032296 21008 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:58:31.034617 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21008
I20250624 07:58:31.035123 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:58:31.043044 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:35921
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:31.072830 21137 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:58:31.073354 21137 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:31.074579 21137 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:58:31.077593 20949 ts_manager.cc:194] Registered new tserver with Master: 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225)
I20250624 07:58:31.080407 20949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:46497
W20250624 07:58:31.350066 21141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:31.350584 21141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:31.351111 21141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:31.386955 21141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:31.387801 21141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:58:31.422469 21141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:31.423777 21141 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:31.425391 21141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:31.441394 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:32.084389 21137 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
W20250624 07:58:31.441450 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:31.442658 21141 server_base.cc:1048] running on GCE node
W20250624 07:58:31.441713 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:32.599428 21141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:32.602195 21141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:32.603619 21141 hybrid_clock.cc:648] HybridClock initialized: now 1750751912603581 us; error 56 us; skew 500 ppm
I20250624 07:58:32.604394 21141 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:32.610493 21141 webserver.cc:469] Webserver started at http://127.17.219.194:34685/ using document root <none> and password file <none>
I20250624 07:58:32.611388 21141 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:32.611578 21141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:32.611989 21141 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:32.616402 21141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9086f0567846438fad679c6b89da2f70"
format_stamp: "Formatted at 2025-06-24 07:58:32 on dist-test-slave-qhgl"
I20250624 07:58:32.617432 21141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9086f0567846438fad679c6b89da2f70"
format_stamp: "Formatted at 2025-06-24 07:58:32 on dist-test-slave-qhgl"
I20250624 07:58:32.624696 21141 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20250624 07:58:32.630198 21158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:32.631224 21141 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:58:32.631502 21141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9086f0567846438fad679c6b89da2f70"
format_stamp: "Formatted at 2025-06-24 07:58:32 on dist-test-slave-qhgl"
I20250624 07:58:32.631798 21141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:32.686172 21141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:32.687604 21141 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:32.687994 21141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:32.690359 21141 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:32.694273 21141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:32.694463 21141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:32.694739 21141 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:32.694896 21141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250624 07:58:32.825939 21141 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:42491
I20250624 07:58:32.826040 21270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:42491 every 8 connection(s)
I20250624 07:58:32.828442 21141 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:58:32.837915 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21141
I20250624 07:58:32.838502 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 07:58:32.844832 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:35921
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:32.849113 21271 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:58:32.849519 21271 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:32.850459 21271 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:58:32.852468 20949 ts_manager.cc:194] Registered new tserver with Master: 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491)
I20250624 07:58:32.853670 20949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:47459
W20250624 07:58:33.147431 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:33.147919 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:33.148425 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:33.179404 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:33.180249 21275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:58:33.214633 21275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:33.216001 21275 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:33.217578 21275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:33.232751 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:33.856686 21271 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
W20250624 07:58:33.232990 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:33.234447 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:34.408063 21283 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:58:34.408078 21275 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:34.413347 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:34.416014 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:34.417423 21275 hybrid_clock.cc:648] HybridClock initialized: now 1750751914417349 us; error 86 us; skew 500 ppm
I20250624 07:58:34.418387 21275 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:34.429462 21275 webserver.cc:469] Webserver started at http://127.17.219.195:37537/ using document root <none> and password file <none>
I20250624 07:58:34.430531 21275 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:34.430788 21275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:34.431351 21275 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:58:34.436478 21275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9abb301874ab40d1b459bd027b1ec047"
format_stamp: "Formatted at 2025-06-24 07:58:34 on dist-test-slave-qhgl"
I20250624 07:58:34.437841 21275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9abb301874ab40d1b459bd027b1ec047"
format_stamp: "Formatted at 2025-06-24 07:58:34 on dist-test-slave-qhgl"
I20250624 07:58:34.446393 21275 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20250624 07:58:34.453559 21291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:34.454878 21275 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.004s
I20250624 07:58:34.455252 21275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9abb301874ab40d1b459bd027b1ec047"
format_stamp: "Formatted at 2025-06-24 07:58:34 on dist-test-slave-qhgl"
I20250624 07:58:34.455580 21275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:34.506414 21275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:34.508198 21275 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:34.508677 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:34.511653 21275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:34.516335 21275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:58:34.516538 21275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:34.516813 21275 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:58:34.516990 21275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:34.676476 21275 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:35163
I20250624 07:58:34.676607 21403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:35163 every 8 connection(s)
I20250624 07:58:34.679391 21275 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:58:34.688640 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21275
I20250624 07:58:34.689075 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 07:58:34.705760 21404 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:58:34.706447 21404 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:34.707671 21404 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:58:34.710253 20948 ts_manager.cc:194] Registered new tserver with Master: 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
I20250624 07:58:34.711953 20948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:57667
I20250624 07:58:34.725666 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:34.765619 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:34.766005 18287 test_util.cc:276] Using random seed: -1309135199
I20250624 07:58:34.808692 20948 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:58:34.854169 21072 tablet_service.cc:1468] Processing CreateTablet for tablet 01ce0df4c62349adb0ca6fe033243cb5 (DEFAULT_TABLE table=TestTable [id=6a3535c3b2364dcebe7899ce83f472d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:34.855684 21072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ce0df4c62349adb0ca6fe033243cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:34.876345 21424 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap starting.
I20250624 07:58:34.881549 21424 tablet_bootstrap.cc:654] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:34.883260 21424 log.cc:826] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:34.888168 21424 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: No bootstrap required, opened a new log
I20250624 07:58:34.888556 21424 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250624 07:58:34.905777 21424 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:34.906353 21424 raft_consensus.cc:383] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:34.906579 21424 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Initialized, Role: FOLLOWER
I20250624 07:58:34.907330 21424 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:34.907842 21424 raft_consensus.cc:397] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:34.908104 21424 raft_consensus.cc:491] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:34.908390 21424 raft_consensus.cc:3058] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:34.913322 21424 raft_consensus.cc:513] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:34.914000 21424 leader_election.cc:304] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318; no voters:
I20250624 07:58:34.915884 21424 leader_election.cc:290] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:58:34.916359 21426 raft_consensus.cc:2802] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:34.919639 21424 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Time spent starting tablet: real 0.031s user 0.024s sys 0.008s
I20250624 07:58:34.919063 21426 raft_consensus.cc:695] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 LEADER]: Becoming Leader. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Running, Role: LEADER
I20250624 07:58:34.920529 21426 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:34.935940 20948 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73f2526ee2e54386be70659b58fe5318 (127.17.219.193). New cstate: current_term: 1 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } }
I20250624 07:58:35.117216 18287 test_util.cc:276] Using random seed: -1308784003
I20250624 07:58:35.140261 20947 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45496:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:58:35.172197 21206 tablet_service.cc:1468] Processing CreateTablet for tablet b87fe1dc267048d2b5403fe74189e17a (DEFAULT_TABLE table=TestTable1 [id=92a288064d7749daa2f0c73b33ea09db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:35.174026 21206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87fe1dc267048d2b5403fe74189e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:35.197764 21444 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap starting.
I20250624 07:58:35.203996 21444 tablet_bootstrap.cc:654] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:35.206135 21444 log.cc:826] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:35.211452 21444 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: No bootstrap required, opened a new log
I20250624 07:58:35.211876 21444 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20250624 07:58:35.230508 21444 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:35.231382 21444 raft_consensus.cc:383] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:35.231688 21444 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Initialized, Role: FOLLOWER
I20250624 07:58:35.232407 21444 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:35.232899 21444 raft_consensus.cc:397] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:35.233191 21444 raft_consensus.cc:491] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:35.233464 21444 raft_consensus.cc:3058] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:35.237967 21444 raft_consensus.cc:513] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:35.238600 21444 leader_election.cc:304] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9086f0567846438fad679c6b89da2f70; no voters:
I20250624 07:58:35.240653 21444 leader_election.cc:290] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:58:35.241057 21446 raft_consensus.cc:2802] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:35.243871 21446 raft_consensus.cc:695] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 LEADER]: Becoming Leader. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Running, Role: LEADER
I20250624 07:58:35.244949 21446 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:35.245457 21444 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Time spent starting tablet: real 0.033s user 0.029s sys 0.004s
I20250624 07:58:35.257431 20947 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9086f0567846438fad679c6b89da2f70 (127.17.219.194). New cstate: current_term: 1 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } health_report { overall_health: HEALTHY } } }
I20250624 07:58:35.482561 18287 test_util.cc:276] Using random seed: -1308418662
I20250624 07:58:35.505280 20948 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45506:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:58:35.536825 21339 tablet_service.cc:1468] Processing CreateTablet for tablet a56578d5b15841ffadd4b990951c28e1 (DEFAULT_TABLE table=TestTable2 [id=8834c5f6ef084da29b950c9f828e4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:58:35.538844 21339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56578d5b15841ffadd4b990951c28e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:35.560019 21465 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:58:35.566170 21465 tablet_bootstrap.cc:654] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Neither blocks nor log segments found. Creating new log.
I20250624 07:58:35.568176 21465 log.cc:826] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:35.573601 21465 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: No bootstrap required, opened a new log
I20250624 07:58:35.574146 21465 ts_tablet_manager.cc:1397] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20250624 07:58:35.592998 21465 raft_consensus.cc:357] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:35.593712 21465 raft_consensus.cc:383] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:58:35.593959 21465 raft_consensus.cc:738] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: FOLLOWER
I20250624 07:58:35.594838 21465 consensus_queue.cc:260] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:35.595481 21465 raft_consensus.cc:397] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:35.595755 21465 raft_consensus.cc:491] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:35.596091 21465 raft_consensus.cc:3058] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:58:35.600580 21465 raft_consensus.cc:513] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:35.601263 21465 leader_election.cc:304] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abb301874ab40d1b459bd027b1ec047; no voters:
I20250624 07:58:35.603171 21465 leader_election.cc:290] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:58:35.603724 21467 raft_consensus.cc:2802] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:58:35.606348 21467 raft_consensus.cc:695] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 LEADER]: Becoming Leader. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Running, Role: LEADER
I20250624 07:58:35.606750 21404 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
I20250624 07:58:35.607543 21467 consensus_queue.cc:237] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:35.608112 21465 ts_tablet_manager.cc:1428] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.034s user 0.031s sys 0.005s
I20250624 07:58:35.620590 20948 catalog_manager.cc:5582] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195). New cstate: current_term: 1 leader_uuid: "9abb301874ab40d1b459bd027b1ec047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } health_report { overall_health: HEALTHY } } }
I20250624 07:58:35.848548 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 20916
W20250624 07:58:35.952322 21137 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:35921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:35921: connect: Connection refused (error 111)
W20250624 07:58:36.273155 21271 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:35921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:35921: connect: Connection refused (error 111)
W20250624 07:58:36.637907 21404 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:35921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:35921: connect: Connection refused (error 111)
I20250624 07:58:40.653434 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21008
I20250624 07:58:40.678444 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21141
I20250624 07:58:40.709782 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21275
I20250624 07:58:40.743181 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:35921
--webserver_interface=127.17.219.254
--webserver_port=45265
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:35921 with env {}
W20250624 07:58:41.088649 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:41.089383 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:41.089887 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:41.123189 21544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:58:41.123526 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:41.123788 21544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:58:41.124032 21544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:58:41.159529 21544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:35921
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=45265
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:41.160967 21544 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:41.162698 21544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:41.179011 21551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:41.183116 21550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:41.187345 21553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:42.371520 21552 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:58:42.371556 21544 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:42.376819 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:42.379748 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:42.381115 21544 hybrid_clock.cc:648] HybridClock initialized: now 1750751922381065 us; error 67 us; skew 500 ppm
I20250624 07:58:42.381954 21544 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:42.389707 21544 webserver.cc:469] Webserver started at http://127.17.219.254:45265/ using document root <none> and password file <none>
I20250624 07:58:42.390673 21544 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:42.390934 21544 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:42.399428 21544 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250624 07:58:42.404392 21561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:42.405519 21544 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 07:58:42.405839 21544 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "58dfc8c4b9e0494280f8f9e271bd4459"
format_stamp: "Formatted at 2025-06-24 07:58:28 on dist-test-slave-qhgl"
I20250624 07:58:42.407935 21544 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:42.459173 21544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:42.460988 21544 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:42.461480 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:42.543819 21544 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:35921
I20250624 07:58:42.543936 21612 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:35921 every 8 connection(s)
I20250624 07:58:42.547005 21544 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:58:42.552841 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21544
I20250624 07:58:42.554731 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:41225
--local_ip_for_outbound_sockets=127.17.219.193
--tserver_master_addrs=127.17.219.254:35921
--webserver_port=46399
--webserver_interface=127.17.219.193
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:42.557619 21613 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:58:42.562232 21613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap starting.
I20250624 07:58:42.576128 21613 log.cc:826] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:42.640414 21613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:42.641454 21613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap complete.
I20250624 07:58:42.672179 21613 raft_consensus.cc:357] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:42.675318 21613 raft_consensus.cc:738] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Initialized, Role: FOLLOWER
I20250624 07:58:42.676440 21613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:42.677292 21613 raft_consensus.cc:397] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:42.677711 21613 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:42.678161 21613 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:58:42.688060 21613 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:42.688980 21613 leader_election.cc:304] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dfc8c4b9e0494280f8f9e271bd4459; no voters:
I20250624 07:58:42.691308 21613 leader_election.cc:290] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250624 07:58:42.691848 21617 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 07:58:42.695801 21617 raft_consensus.cc:695] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 LEADER]: Becoming Leader. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Running, Role: LEADER
I20250624 07:58:42.696786 21613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:58:42.696709 21617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:42.709408 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dfc8c4b9e0494280f8f9e271bd4459. Latest consensus state: current_term: 3 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:42.710568 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:42.717456 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:42.718022 21624 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:58:42.718144 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:42.736135 21624 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=8834c5f6ef084da29b950c9f828e4615]
I20250624 07:58:42.738237 21624 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c7e08e0bdd7c473d922bef6141850a5a]
I20250624 07:58:42.740278 21624 catalog_manager.cc:671] Loaded metadata for table TestTable [id=cba9937b7cb94965b01f4f349650dc9d]
I20250624 07:58:42.751335 21624 tablet_loader.cc:96] loaded metadata for tablet 01ce0df4c62349adb0ca6fe033243cb5 (table TestTable [id=cba9937b7cb94965b01f4f349650dc9d])
I20250624 07:58:42.753440 21624 tablet_loader.cc:96] loaded metadata for tablet a56578d5b15841ffadd4b990951c28e1 (table TestTable2 [id=8834c5f6ef084da29b950c9f828e4615])
I20250624 07:58:42.755125 21624 tablet_loader.cc:96] loaded metadata for tablet b87fe1dc267048d2b5403fe74189e17a (table TestTable1 [id=c7e08e0bdd7c473d922bef6141850a5a])
I20250624 07:58:42.756628 21624 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:58:42.763724 21624 catalog_manager.cc:1261] Loaded cluster ID: f025185adc7b4d40a894b43d1ebfec9b
I20250624 07:58:42.764122 21624 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:58:42.775411 21624 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:58:42.783955 21624 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Loaded TSK: 0
I20250624 07:58:42.786296 21624 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 07:58:42.964546 21615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:42.965046 21615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:42.965567 21615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:42.997486 21615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:42.998427 21615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:58:43.032727 21615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:41225
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=46399
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:43.034044 21615 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:43.035670 21615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:43.053561 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:43.054440 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:43.054215 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:44.249065 21642 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:58:44.249645 21615 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:44.254228 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:44.257066 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:44.258569 21615 hybrid_clock.cc:648] HybridClock initialized: now 1750751924258516 us; error 52 us; skew 500 ppm
I20250624 07:58:44.259686 21615 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:44.267496 21615 webserver.cc:469] Webserver started at http://127.17.219.193:46399/ using document root <none> and password file <none>
I20250624 07:58:44.268788 21615 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:44.269104 21615 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:44.279873 21615 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250624 07:58:44.285734 21650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:44.286970 21615 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250624 07:58:44.287369 21615 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73f2526ee2e54386be70659b58fe5318"
format_stamp: "Formatted at 2025-06-24 07:58:30 on dist-test-slave-qhgl"
I20250624 07:58:44.290050 21615 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:44.353856 21615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:44.355448 21615 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:44.355958 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:44.358546 21615 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:44.364976 21657 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 07:58:44.375864 21615 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 07:58:44.376127 21615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250624 07:58:44.376396 21615 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 07:58:44.381341 21615 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 07:58:44.381553 21615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250624 07:58:44.382054 21657 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap starting.
I20250624 07:58:44.453330 21657 log.cc:826] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:44.597618 21657 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:44.598730 21657 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap complete.
I20250624 07:58:44.600512 21657 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Time spent bootstrapping tablet: real 0.219s user 0.172s sys 0.040s
I20250624 07:58:44.601007 21615 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:41225
I20250624 07:58:44.601194 21764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:41225 every 8 connection(s)
I20250624 07:58:44.603756 21615 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:58:44.606002 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21615
I20250624 07:58:44.608803 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:42491
--local_ip_for_outbound_sockets=127.17.219.194
--tserver_master_addrs=127.17.219.254:35921
--webserver_port=34685
--webserver_interface=127.17.219.194
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:44.622617 21657 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:44.626012 21657 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Initialized, Role: FOLLOWER
I20250624 07:58:44.627579 21657 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:44.628487 21657 raft_consensus.cc:397] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:44.628940 21657 raft_consensus.cc:491] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:44.629448 21657 raft_consensus.cc:3058] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:58:44.637832 21657 raft_consensus.cc:513] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:44.638691 21657 leader_election.cc:304] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318; no voters:
I20250624 07:58:44.651296 21657 leader_election.cc:290] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 07:58:44.652026 21770 raft_consensus.cc:2802] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:58:44.659267 21770 raft_consensus.cc:695] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 LEADER]: Becoming Leader. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Running, Role: LEADER
I20250624 07:58:44.660266 21770 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:58:44.667362 21657 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Time spent starting tablet: real 0.067s user 0.034s sys 0.022s
I20250624 07:58:44.678803 21765 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:58:44.679388 21765 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:44.681049 21765 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:58:44.690310 21578 ts_manager.cc:194] Registered new tserver with Master: 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225)
I20250624 07:58:44.693306 21578 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: term changed from 0 to 2, leader changed from <none> to 73f2526ee2e54386be70659b58fe5318 (127.17.219.193), VOTER 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) added. New cstate: current_term: 2 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } }
W20250624 07:58:44.733940 21578 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 01ce0df4c62349adb0ca6fe033243cb5 with cas_config_opid_index -1: no extra replica candidate found for tablet 01ce0df4c62349adb0ca6fe033243cb5 (table TestTable [id=cba9937b7cb94965b01f4f349650dc9d]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250624 07:58:44.736292 21578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:46697
I20250624 07:58:44.740646 21765 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
W20250624 07:58:44.968243 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:44.968729 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:44.969177 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:45.000967 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:45.001802 21766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:58:45.037286 21766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:42491
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=34685
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:45.038586 21766 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:45.040172 21766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:45.055641 21785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:45.057909 21766 server_base.cc:1048] running on GCE node
W20250624 07:58:45.055922 21787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:45.055879 21784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:46.260085 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:46.262882 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:46.264315 21766 hybrid_clock.cc:648] HybridClock initialized: now 1750751926264270 us; error 67 us; skew 500 ppm
I20250624 07:58:46.265100 21766 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:46.274673 21766 webserver.cc:469] Webserver started at http://127.17.219.194:34685/ using document root <none> and password file <none>
I20250624 07:58:46.275611 21766 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:46.275821 21766 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:46.284152 21766 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250624 07:58:46.288869 21794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:46.290079 21766 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250624 07:58:46.290525 21766 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9086f0567846438fad679c6b89da2f70"
format_stamp: "Formatted at 2025-06-24 07:58:32 on dist-test-slave-qhgl"
I20250624 07:58:46.292963 21766 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:46.341583 21766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:46.343108 21766 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:46.343751 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:46.346310 21766 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:46.351979 21801 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 07:58:46.360497 21766 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 07:58:46.360800 21766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250624 07:58:46.361090 21766 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 07:58:46.366068 21766 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 07:58:46.366271 21766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250624 07:58:46.366844 21801 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap starting.
I20250624 07:58:46.429261 21801 log.cc:826] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:46.516911 21801 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:46.517796 21801 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap complete.
I20250624 07:58:46.519779 21801 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Time spent bootstrapping tablet: real 0.153s user 0.104s sys 0.045s
I20250624 07:58:46.537621 21801 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:46.540081 21801 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Initialized, Role: FOLLOWER
I20250624 07:58:46.541040 21801 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:46.541749 21801 raft_consensus.cc:397] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:46.542219 21801 raft_consensus.cc:491] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:46.542636 21801 raft_consensus.cc:3058] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:58:46.552052 21801 raft_consensus.cc:513] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:46.553058 21801 leader_election.cc:304] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9086f0567846438fad679c6b89da2f70; no voters:
I20250624 07:58:46.557238 21801 leader_election.cc:290] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 07:58:46.557736 21902 raft_consensus.cc:2802] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:58:46.564949 21902 raft_consensus.cc:695] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 LEADER]: Becoming Leader. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Running, Role: LEADER
I20250624 07:58:46.566045 21902 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } }
I20250624 07:58:46.569355 21801 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Time spent starting tablet: real 0.049s user 0.042s sys 0.005s
I20250624 07:58:46.581081 21766 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:42491
I20250624 07:58:46.581632 21913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:42491 every 8 connection(s)
I20250624 07:58:46.583916 21766 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:58:46.587759 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21766
I20250624 07:58:46.589602 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:35163
--local_ip_for_outbound_sockets=127.17.219.195
--tserver_master_addrs=127.17.219.254:35921
--webserver_port=37537
--webserver_interface=127.17.219.195
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:46.603166 21914 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:58:46.603581 21914 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:46.604524 21914 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:58:46.607731 21578 ts_manager.cc:194] Registered new tserver with Master: 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491)
I20250624 07:58:46.608681 21578 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 reported cstate change: term changed from 0 to 2, leader changed from <none> to 9086f0567846438fad679c6b89da2f70 (127.17.219.194), VOTER 9086f0567846438fad679c6b89da2f70 (127.17.219.194) added. New cstate: current_term: 2 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } health_report { overall_health: HEALTHY } } }
I20250624 07:58:46.620760 21578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:44453
I20250624 07:58:46.626430 21914 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
I20250624 07:58:46.640736 21864 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: true } }
I20250624 07:58:46.644944 21904 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: true } } }
I20250624 07:58:46.653874 21565 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b87fe1dc267048d2b5403fe74189e17a with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250624 07:58:46.657429 21578 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 reported cstate change: config changed from index -1 to 9, NON_VOTER 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) added. New cstate: current_term: 2 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250624 07:58:46.667369 21797 consensus_peers.cc:487] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 -> Peer 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225): Couldn't send request to peer 73f2526ee2e54386be70659b58fe5318. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b87fe1dc267048d2b5403fe74189e17a. This is attempt 1: this message will repeat every 5th retry.
W20250624 07:58:46.924654 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:46.925171 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:46.925716 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:46.957989 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:46.958951 21918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:58:46.994119 21918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:35163
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=37537
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:46.995481 21918 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:46.997082 21918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:47.014923 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:47.044102 21720 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: NON_VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: true } }
I20250624 07:58:47.051133 21939 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 9086f0567846438fad679c6b89da2f70 (127.17.219.194) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: NON_VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: true } } }
I20250624 07:58:47.059718 21564 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 01ce0df4c62349adb0ca6fe033243cb5 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250624 07:58:47.063220 21578 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index -1 to 8, NON_VOTER 9086f0567846438fad679c6b89da2f70 (127.17.219.194) added. New cstate: current_term: 2 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: NON_VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250624 07:58:47.071028 21654 consensus_peers.cc:487] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 -> Peer 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491): Couldn't send request to peer 9086f0567846438fad679c6b89da2f70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01ce0df4c62349adb0ca6fe033243cb5. This is attempt 1: this message will repeat every 5th retry.
I20250624 07:58:47.257459 21942 ts_tablet_manager.cc:927] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Initiating tablet copy from peer 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491)
I20250624 07:58:47.260475 21942 tablet_copy_client.cc:323] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: tablet copy: Beginning tablet copy session from remote peer at address 127.17.219.194:42491
I20250624 07:58:47.262733 21884 tablet_copy_service.cc:140] P 9086f0567846438fad679c6b89da2f70: Received BeginTabletCopySession request for tablet b87fe1dc267048d2b5403fe74189e17a from peer 73f2526ee2e54386be70659b58fe5318 ({username='slave'} at 127.17.219.193:38075)
I20250624 07:58:47.263219 21884 tablet_copy_service.cc:161] P 9086f0567846438fad679c6b89da2f70: Beginning new tablet copy session on tablet b87fe1dc267048d2b5403fe74189e17a from peer 73f2526ee2e54386be70659b58fe5318 at {username='slave'} at 127.17.219.193:38075: session id = 73f2526ee2e54386be70659b58fe5318-b87fe1dc267048d2b5403fe74189e17a
I20250624 07:58:47.268138 21884 tablet_copy_source_session.cc:215] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 07:58:47.273459 21942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87fe1dc267048d2b5403fe74189e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:47.292548 21942 tablet_copy_client.cc:806] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: tablet copy: Starting download of 0 data blocks...
I20250624 07:58:47.293077 21942 tablet_copy_client.cc:670] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: tablet copy: Starting download of 1 WAL segments...
I20250624 07:58:47.299321 21942 tablet_copy_client.cc:538] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 07:58:47.305306 21942 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Bootstrap starting.
I20250624 07:58:47.389803 21942 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:47.390527 21942 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Bootstrap complete.
I20250624 07:58:47.391016 21942 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Time spent bootstrapping tablet: real 0.086s user 0.076s sys 0.012s
I20250624 07:58:47.392915 21942 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: true } }
I20250624 07:58:47.393374 21942 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Initialized, Role: LEARNER
I20250624 07:58:47.393803 21942 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: NON_VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: true } }
I20250624 07:58:47.395226 21942 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 07:58:47.397481 21884 tablet_copy_service.cc:342] P 9086f0567846438fad679c6b89da2f70: Request end of tablet copy session 73f2526ee2e54386be70659b58fe5318-b87fe1dc267048d2b5403fe74189e17a received from {username='slave'} at 127.17.219.193:38075
I20250624 07:58:47.397960 21884 tablet_copy_service.cc:434] P 9086f0567846438fad679c6b89da2f70: ending tablet copy session 73f2526ee2e54386be70659b58fe5318-b87fe1dc267048d2b5403fe74189e17a on tablet b87fe1dc267048d2b5403fe74189e17a with peer 73f2526ee2e54386be70659b58fe5318
I20250624 07:58:47.476660 21945 ts_tablet_manager.cc:927] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Initiating tablet copy from peer 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225)
I20250624 07:58:47.479494 21945 tablet_copy_client.cc:323] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: tablet copy: Beginning tablet copy session from remote peer at address 127.17.219.193:41225
I20250624 07:58:47.481727 21740 tablet_copy_service.cc:140] P 73f2526ee2e54386be70659b58fe5318: Received BeginTabletCopySession request for tablet 01ce0df4c62349adb0ca6fe033243cb5 from peer 9086f0567846438fad679c6b89da2f70 ({username='slave'} at 127.17.219.194:40119)
I20250624 07:58:47.482286 21740 tablet_copy_service.cc:161] P 73f2526ee2e54386be70659b58fe5318: Beginning new tablet copy session on tablet 01ce0df4c62349adb0ca6fe033243cb5 from peer 9086f0567846438fad679c6b89da2f70 at {username='slave'} at 127.17.219.194:40119: session id = 9086f0567846438fad679c6b89da2f70-01ce0df4c62349adb0ca6fe033243cb5
I20250624 07:58:47.487826 21740 tablet_copy_source_session.cc:215] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 07:58:47.491760 21945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ce0df4c62349adb0ca6fe033243cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:47.505435 21945 tablet_copy_client.cc:806] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: tablet copy: Starting download of 0 data blocks...
I20250624 07:58:47.506026 21945 tablet_copy_client.cc:670] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: tablet copy: Starting download of 1 WAL segments...
I20250624 07:58:47.510883 21945 tablet_copy_client.cc:538] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 07:58:47.517304 21945 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Bootstrap starting.
I20250624 07:58:47.612140 21945 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:47.613144 21945 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Bootstrap complete.
I20250624 07:58:47.613809 21945 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Time spent bootstrapping tablet: real 0.097s user 0.075s sys 0.021s
I20250624 07:58:47.616545 21945 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: NON_VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: true } }
I20250624 07:58:47.617238 21945 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Initialized, Role: LEARNER
I20250624 07:58:47.617846 21945 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: NON_VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: true } }
I20250624 07:58:47.619874 21945 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250624 07:58:47.623107 21740 tablet_copy_service.cc:342] P 73f2526ee2e54386be70659b58fe5318: Request end of tablet copy session 9086f0567846438fad679c6b89da2f70-01ce0df4c62349adb0ca6fe033243cb5 received from {username='slave'} at 127.17.219.194:40119
I20250624 07:58:47.623660 21740 tablet_copy_service.cc:434] P 73f2526ee2e54386be70659b58fe5318: ending tablet copy session 9086f0567846438fad679c6b89da2f70-01ce0df4c62349adb0ca6fe033243cb5 on tablet 01ce0df4c62349adb0ca6fe033243cb5 with peer 9086f0567846438fad679c6b89da2f70
I20250624 07:58:47.815943 21720 raft_consensus.cc:1215] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250624 07:58:48.078389 21864 raft_consensus.cc:1215] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250624 07:58:47.015039 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:47.017445 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:48.243280 21933 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:58:48.243351 21918 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:58:48.248266 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:48.251152 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:48.252585 21918 hybrid_clock.cc:648] HybridClock initialized: now 1750751928252530 us; error 75 us; skew 500 ppm
I20250624 07:58:48.253414 21918 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:48.254317 21948 raft_consensus.cc:1062] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: attempting to promote NON_VOTER 73f2526ee2e54386be70659b58fe5318 to VOTER
I20250624 07:58:48.255719 21948 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } }
I20250624 07:58:48.260257 21720 raft_consensus.cc:1273] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 LEARNER]: Refusing update from remote peer 9086f0567846438fad679c6b89da2f70: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250624 07:58:48.261391 21918 webserver.cc:469] Webserver started at http://127.17.219.195:37537/ using document root <none> and password file <none>
I20250624 07:58:48.261466 21948 consensus_queue.cc:1035] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250624 07:58:48.263134 21918 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:48.263478 21918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:48.269249 21949 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } }
I20250624 07:58:48.271030 21720 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } }
I20250624 07:58:48.274703 21918 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20250624 07:58:48.281315 21958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:48.283140 21918 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20250624 07:58:48.283221 21577 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 reported cstate change: config changed from index 9 to 10, 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250624 07:58:48.283686 21918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9abb301874ab40d1b459bd027b1ec047"
format_stamp: "Formatted at 2025-06-24 07:58:34 on dist-test-slave-qhgl"
I20250624 07:58:48.286685 21918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:48.360283 21918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:48.362002 21918 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:48.362493 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:48.365418 21918 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:58:48.371754 21969 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 07:58:48.379652 21918 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 07:58:48.379900 21918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250624 07:58:48.380240 21918 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 07:58:48.385452 21918 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 07:58:48.385635 21918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.003s
I20250624 07:58:48.386211 21969 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:58:48.448474 21969 log.cc:826] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:48.496114 21951 raft_consensus.cc:1062] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: attempting to promote NON_VOTER 9086f0567846438fad679c6b89da2f70 to VOTER
I20250624 07:58:48.497969 21951 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } }
I20250624 07:58:48.505226 21864 raft_consensus.cc:1273] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 LEARNER]: Refusing update from remote peer 73f2526ee2e54386be70659b58fe5318: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250624 07:58:48.507345 21951 consensus_queue.cc:1035] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250624 07:58:48.517597 21950 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9086f0567846438fad679c6b89da2f70 (127.17.219.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } }
I20250624 07:58:48.522568 21864 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9086f0567846438fad679c6b89da2f70 (127.17.219.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } }
I20250624 07:58:48.530611 21577 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index 8 to 9, 9086f0567846438fad679c6b89da2f70 (127.17.219.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250624 07:58:48.569105 21969 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:48.570556 21969 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap complete.
I20250624 07:58:48.572799 21969 ts_tablet_manager.cc:1397] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.187s user 0.125s sys 0.050s
I20250624 07:58:48.603183 21969 raft_consensus.cc:357] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:48.606315 21969 raft_consensus.cc:738] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: FOLLOWER
I20250624 07:58:48.607391 21969 consensus_queue.cc:260] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:48.608034 21969 raft_consensus.cc:397] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:48.608350 21969 raft_consensus.cc:491] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:48.608700 21969 raft_consensus.cc:3058] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:58:48.621328 21969 raft_consensus.cc:513] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:48.622313 21969 leader_election.cc:304] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abb301874ab40d1b459bd027b1ec047; no voters:
I20250624 07:58:48.625128 21969 leader_election.cc:290] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 07:58:48.625404 22054 raft_consensus.cc:2802] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:58:48.628384 22054 raft_consensus.cc:695] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEADER]: Becoming Leader. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Running, Role: LEADER
I20250624 07:58:48.629702 22054 consensus_queue.cc:237] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:58:48.635433 21969 ts_tablet_manager.cc:1428] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.062s user 0.045s sys 0.011s
I20250624 07:58:48.682384 21918 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:35163
I20250624 07:58:48.682893 22083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:35163 every 8 connection(s)
I20250624 07:58:48.685333 21918 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:58:48.686017 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 21918
I20250624 07:58:48.716257 22084 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:58:48.716787 22084 heartbeater.cc:461] Registering TS with master...
I20250624 07:58:48.717950 22084 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:58:48.721607 21578 ts_manager.cc:194] Registered new tserver with Master: 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
I20250624 07:58:48.722762 21578 catalog_manager.cc:5582] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "9abb301874ab40d1b459bd027b1ec047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } health_report { overall_health: HEALTHY } } }
I20250624 07:58:48.730260 21578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:52719
I20250624 07:58:48.732240 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:48.733755 22084 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
I20250624 07:58:48.735497 21720 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } }
I20250624 07:58:48.737417 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:58:48.742412 21864 raft_consensus.cc:1273] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Refusing update from remote peer 73f2526ee2e54386be70659b58fe5318: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20250624 07:58:48.743000 18287 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 01ce0df4c62349adb0ca6fe033243cb5: tablet_id: "01ce0df4c62349adb0ca6fe033243cb5" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250624 07:58:48.743780 21939 consensus_queue.cc:1035] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250624 07:58:48.749828 21950 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } } }
I20250624 07:58:48.751274 21864 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } } }
I20250624 07:58:48.756908 21564 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 01ce0df4c62349adb0ca6fe033243cb5 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
W20250624 07:58:48.761271 21652 consensus_peers.cc:487] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 -> Peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Couldn't send request to peer 9abb301874ab40d1b459bd027b1ec047. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01ce0df4c62349adb0ca6fe033243cb5. This is attempt 1: this message will repeat every 5th retry.
I20250624 07:58:48.760751 21577 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index 9 to 10, NON_VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New cstate: current_term: 2 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250624 07:58:48.861251 21564 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b87fe1dc267048d2b5403fe74189e17a with cas_config_opid_index 9: aborting the task: latest config opid_index 10; task opid_index 9
I20250624 07:58:48.915431 21864 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } }
I20250624 07:58:48.920790 21720 raft_consensus.cc:1273] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Refusing update from remote peer 9086f0567846438fad679c6b89da2f70: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250624 07:58:48.922230 21949 consensus_queue.cc:1035] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250624 07:58:48.928134 22092 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } } }
W20250624 07:58:48.929054 21796 consensus_peers.cc:487] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 -> Peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Couldn't send request to peer 9abb301874ab40d1b459bd027b1ec047. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b87fe1dc267048d2b5403fe74189e17a. This is attempt 1: this message will repeat every 5th retry.
I20250624 07:58:48.929622 21720 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } } }
I20250624 07:58:48.935148 21565 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b87fe1dc267048d2b5403fe74189e17a with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250624 07:58:48.939287 21578 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index 10 to 11, NON_VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New cstate: current_term: 2 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } } }
I20250624 07:58:49.217713 22095 ts_tablet_manager.cc:927] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Initiating tablet copy from peer 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225)
I20250624 07:58:49.219762 22095 tablet_copy_client.cc:323] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Beginning tablet copy session from remote peer at address 127.17.219.193:41225
I20250624 07:58:49.230106 21740 tablet_copy_service.cc:140] P 73f2526ee2e54386be70659b58fe5318: Received BeginTabletCopySession request for tablet 01ce0df4c62349adb0ca6fe033243cb5 from peer 9abb301874ab40d1b459bd027b1ec047 ({username='slave'} at 127.17.219.195:41913)
I20250624 07:58:49.230494 21740 tablet_copy_service.cc:161] P 73f2526ee2e54386be70659b58fe5318: Beginning new tablet copy session on tablet 01ce0df4c62349adb0ca6fe033243cb5 from peer 9abb301874ab40d1b459bd027b1ec047 at {username='slave'} at 127.17.219.195:41913: session id = 9abb301874ab40d1b459bd027b1ec047-01ce0df4c62349adb0ca6fe033243cb5
I20250624 07:58:49.234898 21740 tablet_copy_source_session.cc:215] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 07:58:49.238154 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ce0df4c62349adb0ca6fe033243cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:49.250598 22095 tablet_copy_client.cc:806] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Starting download of 0 data blocks...
I20250624 07:58:49.251091 22095 tablet_copy_client.cc:670] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Starting download of 1 WAL segments...
I20250624 07:58:49.254889 22095 tablet_copy_client.cc:538] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 07:58:49.260310 22095 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:58:49.291860 21562 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 01ce0df4c62349adb0ca6fe033243cb5 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250624 07:58:49.335099 22095 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:49.335922 22095 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap complete.
I20250624 07:58:49.336472 22095 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.076s user 0.060s sys 0.016s
I20250624 07:58:49.338358 22095 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } }
I20250624 07:58:49.338951 22095 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: LEARNER
I20250624 07:58:49.339424 22095 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } }
I20250624 07:58:49.342074 22100 ts_tablet_manager.cc:927] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Initiating tablet copy from peer 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491)
I20250624 07:58:49.342912 22095 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250624 07:58:49.344662 21740 tablet_copy_service.cc:342] P 73f2526ee2e54386be70659b58fe5318: Request end of tablet copy session 9abb301874ab40d1b459bd027b1ec047-01ce0df4c62349adb0ca6fe033243cb5 received from {username='slave'} at 127.17.219.195:41913
I20250624 07:58:49.345064 21740 tablet_copy_service.cc:434] P 73f2526ee2e54386be70659b58fe5318: ending tablet copy session 9abb301874ab40d1b459bd027b1ec047-01ce0df4c62349adb0ca6fe033243cb5 on tablet 01ce0df4c62349adb0ca6fe033243cb5 with peer 9abb301874ab40d1b459bd027b1ec047
I20250624 07:58:49.345173 22100 tablet_copy_client.cc:323] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Beginning tablet copy session from remote peer at address 127.17.219.194:42491
I20250624 07:58:49.356859 21884 tablet_copy_service.cc:140] P 9086f0567846438fad679c6b89da2f70: Received BeginTabletCopySession request for tablet b87fe1dc267048d2b5403fe74189e17a from peer 9abb301874ab40d1b459bd027b1ec047 ({username='slave'} at 127.17.219.195:54335)
I20250624 07:58:49.357288 21884 tablet_copy_service.cc:161] P 9086f0567846438fad679c6b89da2f70: Beginning new tablet copy session on tablet b87fe1dc267048d2b5403fe74189e17a from peer 9abb301874ab40d1b459bd027b1ec047 at {username='slave'} at 127.17.219.195:54335: session id = 9abb301874ab40d1b459bd027b1ec047-b87fe1dc267048d2b5403fe74189e17a
I20250624 07:58:49.361598 21884 tablet_copy_source_session.cc:215] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 07:58:49.364193 22100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87fe1dc267048d2b5403fe74189e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:58:49.374574 22100 tablet_copy_client.cc:806] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Starting download of 0 data blocks...
I20250624 07:58:49.375106 22100 tablet_copy_client.cc:670] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Starting download of 1 WAL segments...
I20250624 07:58:49.379005 22100 tablet_copy_client.cc:538] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 07:58:49.384577 22100 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:58:49.466221 22100 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:49.466903 22100 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Bootstrap complete.
I20250624 07:58:49.467310 22100 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.083s user 0.068s sys 0.016s
I20250624 07:58:49.468685 22100 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } }
I20250624 07:58:49.469152 22100 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: LEARNER
I20250624 07:58:49.469530 22100 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: NON_VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: true } }
I20250624 07:58:49.471117 22100 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 07:58:49.472776 21884 tablet_copy_service.cc:342] P 9086f0567846438fad679c6b89da2f70: Request end of tablet copy session 9abb301874ab40d1b459bd027b1ec047-b87fe1dc267048d2b5403fe74189e17a received from {username='slave'} at 127.17.219.195:54335
I20250624 07:58:49.473094 21884 tablet_copy_service.cc:434] P 9086f0567846438fad679c6b89da2f70: ending tablet copy session 9abb301874ab40d1b459bd027b1ec047-b87fe1dc267048d2b5403fe74189e17a on tablet b87fe1dc267048d2b5403fe74189e17a with peer 9abb301874ab40d1b459bd027b1ec047
I20250624 07:58:49.667009 22034 raft_consensus.cc:1215] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250624 07:58:49.747850 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73f2526ee2e54386be70659b58fe5318 to finish bootstrapping
I20250624 07:58:49.762378 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9086f0567846438fad679c6b89da2f70 to finish bootstrapping
I20250624 07:58:49.774818 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9abb301874ab40d1b459bd027b1ec047 to finish bootstrapping
I20250624 07:58:49.843425 22034 raft_consensus.cc:1215] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250624 07:58:50.071866 21700 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 07:58:50.073931 21844 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 07:58:50.077769 22012 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 07:58:50.210014 22106 raft_consensus.cc:1062] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: attempting to promote NON_VOTER 9abb301874ab40d1b459bd027b1ec047 to VOTER
I20250624 07:58:50.212000 22106 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:58:50.238359 22034 raft_consensus.cc:1273] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Refusing update from remote peer 73f2526ee2e54386be70659b58fe5318: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250624 07:58:50.239097 21864 raft_consensus.cc:1273] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Refusing update from remote peer 73f2526ee2e54386be70659b58fe5318: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250624 07:58:50.240408 22138 consensus_queue.cc:1035] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250624 07:58:50.241062 22106 consensus_queue.cc:1035] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250624 07:58:50.250663 22138 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:58:50.256840 21864 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:58:50.267798 21577 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index 10 to 11, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250624 07:58:50.268772 22034 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
58dfc8c4b9e0494280f8f9e271bd4459 | 127.17.219.254:35921 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.219.212:38505 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
73f2526ee2e54386be70659b58fe5318 | 127.17.219.193:41225 | HEALTHY | <none> | 1 | 0
9086f0567846438fad679c6b89da2f70 | 127.17.219.194:42491 | HEALTHY | <none> | 1 | 0
9abb301874ab40d1b459bd027b1ec047 | 127.17.219.195:35163 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.17.219.193 | experimental | 127.17.219.193:41225
local_ip_for_outbound_sockets | 127.17.219.194 | experimental | 127.17.219.194:42491
local_ip_for_outbound_sockets | 127.17.219.195 | experimental | 127.17.219.195:35163
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.17.219.193:41225
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.17.219.194:42491
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.17.219.195:35163
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.219.212:38505 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 07:58:50.368198 18287 log_verifier.cc:126] Checking tablet 01ce0df4c62349adb0ca6fe033243cb5
I20250624 07:58:50.452139 22118 raft_consensus.cc:1062] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: attempting to promote NON_VOTER 9abb301874ab40d1b459bd027b1ec047 to VOTER
I20250624 07:58:50.453575 22118 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:58:50.459559 22034 raft_consensus.cc:1273] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 LEARNER]: Refusing update from remote peer 9086f0567846438fad679c6b89da2f70: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250624 07:58:50.460855 22147 consensus_queue.cc:1035] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250624 07:58:50.461565 21720 raft_consensus.cc:1273] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Refusing update from remote peer 9086f0567846438fad679c6b89da2f70: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250624 07:58:50.463114 22147 consensus_queue.cc:1035] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250624 07:58:50.468899 22149 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:58:50.470782 22034 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:58:50.472103 18287 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 01ce0df4c62349adb0ca6fe033243cb5
I20250624 07:58:50.472546 18287 log_verifier.cc:126] Checking tablet a56578d5b15841ffadd4b990951c28e1
I20250624 07:58:50.472679 21720 raft_consensus.cc:2953] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:58:50.485100 21578 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 reported cstate change: config changed from index 11 to 12, 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:58:50.510485 18287 log_verifier.cc:177] Verified matching terms for 8 ops in tablet a56578d5b15841ffadd4b990951c28e1
I20250624 07:58:50.510736 18287 log_verifier.cc:126] Checking tablet b87fe1dc267048d2b5403fe74189e17a
I20250624 07:58:50.602934 18287 log_verifier.cc:177] Verified matching terms for 12 ops in tablet b87fe1dc267048d2b5403fe74189e17a
I20250624 07:58:50.603528 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21544
I20250624 07:58:50.641678 18287 minidump.cc:252] Setting minidump size limit to 20M
I20250624 07:58:50.651944 18287 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:50.653545 18287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:50.668630 22153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:50.670254 22154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:50.878065 22156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:50.878201 18287 server_base.cc:1048] running on GCE node
I20250624 07:58:50.879623 18287 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250624 07:58:50.879846 18287 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250624 07:58:50.880023 18287 hybrid_clock.cc:648] HybridClock initialized: now 1750751930880002 us; error 0 us; skew 500 ppm
I20250624 07:58:50.880749 18287 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:50.888198 18287 webserver.cc:469] Webserver started at http://0.0.0.0:36221/ using document root <none> and password file <none>
I20250624 07:58:50.889037 18287 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:50.889240 18287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:50.894548 18287 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250624 07:58:50.898413 22161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:50.899454 18287 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250624 07:58:50.899778 18287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "58dfc8c4b9e0494280f8f9e271bd4459"
format_stamp: "Formatted at 2025-06-24 07:58:28 on dist-test-slave-qhgl"
I20250624 07:58:50.901422 18287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:50.922387 18287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:50.923966 18287 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:50.924414 18287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:50.933010 18287 sys_catalog.cc:263] Verifying existing consensus state
W20250624 07:58:50.936435 18287 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.17.219.254:35921 exists but no master address supplied!
I20250624 07:58:50.938436 18287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap starting.
I20250624 07:58:50.981069 18287 log.cc:826] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:51.057782 18287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:51.058763 18287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap complete.
I20250624 07:58:51.072868 18287 raft_consensus.cc:357] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:51.073431 18287 raft_consensus.cc:738] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Initialized, Role: FOLLOWER
I20250624 07:58:51.074167 18287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:51.074642 18287 raft_consensus.cc:397] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:51.074970 18287 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:51.075291 18287 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 3 FOLLOWER]: Advancing to term 4
I20250624 07:58:51.080554 18287 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:51.081233 18287 leader_election.cc:304] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dfc8c4b9e0494280f8f9e271bd4459; no voters:
I20250624 07:58:51.082422 18287 leader_election.cc:290] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250624 07:58:51.082866 22168 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 4 FOLLOWER]: Leader election won for term 4
I20250624 07:58:51.084353 22168 raft_consensus.cc:695] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 4 LEADER]: Becoming Leader. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Running, Role: LEADER
I20250624 07:58:51.085247 22168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:51.096119 22170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dfc8c4b9e0494280f8f9e271bd4459. Latest consensus state: current_term: 4 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:51.096894 22170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:51.096817 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:51.097466 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:51.124786 18287 tablet_replica.cc:331] stopping tablet replica
I20250624 07:58:51.125541 18287 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 4 LEADER]: Raft consensus shutting down.
I20250624 07:58:51.126088 18287 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250624 07:58:51.128901 18287 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250624 07:58:51.129452 18287 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250624 07:58:51.155560 18287 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250624 07:58:51.499639 22084 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:35921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:35921: connect: Connection refused (error 111)
W20250624 07:58:51.501472 21765 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:35921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:35921: connect: Connection refused (error 111)
W20250624 07:58:51.506078 21914 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:35921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:35921: connect: Connection refused (error 111)
I20250624 07:58:56.401785 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21615
I20250624 07:58:56.429538 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21766
I20250624 07:58:56.457620 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 21918
I20250624 07:58:56.488307 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:35921
--webserver_interface=127.17.219.254
--webserver_port=45265
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:35921 with env {}
W20250624 07:58:56.787242 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:56.787812 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:56.788257 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:56.818246 22242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:58:56.818517 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:56.818835 22242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:58:56.819103 22242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:58:56.852754 22242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:35921
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=45265
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:56.854086 22242 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:56.855700 22242 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:56.869814 22249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:56.870358 22251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:56.870944 22248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:58:56.872527 22242 server_base.cc:1048] running on GCE node
I20250624 07:58:58.055351 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:58:58.057932 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:58:58.059353 22242 hybrid_clock.cc:648] HybridClock initialized: now 1750751938059318 us; error 45 us; skew 500 ppm
I20250624 07:58:58.060122 22242 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:58:58.070902 22242 webserver.cc:469] Webserver started at http://127.17.219.254:45265/ using document root <none> and password file <none>
I20250624 07:58:58.071870 22242 fs_manager.cc:362] Metadata directory not provided
I20250624 07:58:58.072124 22242 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:58:58.079910 22242 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250624 07:58:58.084337 22258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:58:58.085297 22242 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250624 07:58:58.085623 22242 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "58dfc8c4b9e0494280f8f9e271bd4459"
format_stamp: "Formatted at 2025-06-24 07:58:28 on dist-test-slave-qhgl"
I20250624 07:58:58.087685 22242 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:58:58.141671 22242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:58:58.143196 22242 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:58:58.143585 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:58:58.212396 22242 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:35921
I20250624 07:58:58.212514 22309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:35921 every 8 connection(s)
I20250624 07:58:58.215328 22242 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:58:58.222301 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 22242
I20250624 07:58:58.224311 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:41225
--local_ip_for_outbound_sockets=127.17.219.193
--tserver_master_addrs=127.17.219.254:35921
--webserver_port=46399
--webserver_interface=127.17.219.193
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:58:58.228331 22310 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:58:58.236699 22310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap starting.
I20250624 07:58:58.247695 22310 log.cc:826] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Log is configured to *not* fsync() on all Append() calls
I20250624 07:58:58.325747 22310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:58:58.326571 22310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Bootstrap complete.
I20250624 07:58:58.346800 22310 raft_consensus.cc:357] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:58.348810 22310 raft_consensus.cc:738] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Initialized, Role: FOLLOWER
I20250624 07:58:58.349619 22310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:58.350099 22310 raft_consensus.cc:397] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:58:58.350348 22310 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:58:58.350615 22310 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 5 FOLLOWER]: Advancing to term 6
I20250624 07:58:58.355782 22310 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:58.356432 22310 leader_election.cc:304] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dfc8c4b9e0494280f8f9e271bd4459; no voters:
I20250624 07:58:58.358537 22310 leader_election.cc:290] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250624 07:58:58.359071 22314 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 6 FOLLOWER]: Leader election won for term 6
I20250624 07:58:58.361999 22314 raft_consensus.cc:695] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [term 6 LEADER]: Becoming Leader. State: Replica: 58dfc8c4b9e0494280f8f9e271bd4459, State: Running, Role: LEADER
I20250624 07:58:58.363068 22310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:58:58.362851 22314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } }
I20250624 07:58:58.373768 22315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:58.375548 22315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:58.374936 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dfc8c4b9e0494280f8f9e271bd4459. Latest consensus state: current_term: 6 leader_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dfc8c4b9e0494280f8f9e271bd4459" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 35921 } } }
I20250624 07:58:58.377247 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459 [sys.catalog]: This master's current role is: LEADER
I20250624 07:58:58.383533 22322 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:58:58.396947 22322 catalog_manager.cc:671] Loaded metadata for table TestTable [id=29fa83bc689b4c45b26be3f7ac0f727c]
I20250624 07:58:58.399984 22322 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=8834c5f6ef084da29b950c9f828e4615]
I20250624 07:58:58.401666 22322 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c7e08e0bdd7c473d922bef6141850a5a]
I20250624 07:58:58.410568 22322 tablet_loader.cc:96] loaded metadata for tablet 01ce0df4c62349adb0ca6fe033243cb5 (table TestTable [id=29fa83bc689b4c45b26be3f7ac0f727c])
I20250624 07:58:58.412002 22322 tablet_loader.cc:96] loaded metadata for tablet a56578d5b15841ffadd4b990951c28e1 (table TestTable2 [id=8834c5f6ef084da29b950c9f828e4615])
I20250624 07:58:58.414850 22322 tablet_loader.cc:96] loaded metadata for tablet b87fe1dc267048d2b5403fe74189e17a (table TestTable1 [id=c7e08e0bdd7c473d922bef6141850a5a])
I20250624 07:58:58.416263 22322 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:58:58.421830 22322 catalog_manager.cc:1261] Loaded cluster ID: f025185adc7b4d40a894b43d1ebfec9b
I20250624 07:58:58.422199 22322 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:58:58.430862 22322 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:58:58.437371 22322 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 58dfc8c4b9e0494280f8f9e271bd4459: Loaded TSK: 0
I20250624 07:58:58.439092 22322 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 07:58:58.574503 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:58:58.575086 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:58:58.575590 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:58:58.606665 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:58:58.607573 22312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:58:58.642522 22312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:41225
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=46399
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:58:58.643877 22312 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:58:58.645439 22312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:58:58.662777 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:58.666898 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:58:58.663004 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:00.124677 22339 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1459 milliseconds
I20250624 07:59:00.124814 22312 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:00.126235 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:00.128849 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:00.130316 22312 hybrid_clock.cc:648] HybridClock initialized: now 1750751940130267 us; error 51 us; skew 500 ppm
I20250624 07:59:00.131349 22312 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:00.138859 22312 webserver.cc:469] Webserver started at http://127.17.219.193:46399/ using document root <none> and password file <none>
I20250624 07:59:00.140057 22312 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:00.140332 22312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:00.150686 22312 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250624 07:59:00.156569 22347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:00.157711 22312 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250624 07:59:00.158098 22312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73f2526ee2e54386be70659b58fe5318"
format_stamp: "Formatted at 2025-06-24 07:58:30 on dist-test-slave-qhgl"
I20250624 07:59:00.160808 22312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:00.261207 22312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:00.262938 22312 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:00.263418 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:00.266353 22312 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:00.272859 22354 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250624 07:59:00.290863 22312 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250624 07:59:00.291126 22312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.000s sys 0.002s
I20250624 07:59:00.291409 22312 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250624 07:59:00.296887 22354 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap starting.
I20250624 07:59:00.299415 22312 ts_tablet_manager.cc:610] Registered 2 tablets
I20250624 07:59:00.299635 22312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.003s sys 0.005s
I20250624 07:59:00.363750 22354 log.cc:826] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:00.472718 22354 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:00.473688 22354 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Bootstrap complete.
I20250624 07:59:00.475489 22354 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Time spent bootstrapping tablet: real 0.179s user 0.138s sys 0.039s
I20250624 07:59:00.497685 22354 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:00.501068 22354 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Initialized, Role: FOLLOWER
I20250624 07:59:00.502153 22354 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:00.506558 22354 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318: Time spent starting tablet: real 0.031s user 0.029s sys 0.000s
I20250624 07:59:00.507462 22354 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Bootstrap starting.
I20250624 07:59:00.523406 22312 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:41225
I20250624 07:59:00.523730 22462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:41225 every 8 connection(s)
I20250624 07:59:00.526580 22312 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:59:00.539417 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 22312
I20250624 07:59:00.542016 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:42491
--local_ip_for_outbound_sockets=127.17.219.194
--tserver_master_addrs=127.17.219.254:35921
--webserver_port=34685
--webserver_interface=127.17.219.194
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:59:00.580212 22463 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:59:00.580739 22463 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:00.582015 22463 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:59:00.587642 22275 ts_manager.cc:194] Registered new tserver with Master: 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225)
I20250624 07:59:00.592741 22275 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 73f2526ee2e54386be70659b58fe5318 (127.17.219.193) added, VOTER 9086f0567846438fad679c6b89da2f70 (127.17.219.194) added, VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } } }
I20250624 07:59:00.668008 22275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:55463
I20250624 07:59:00.676026 22354 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:00.676934 22354 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Bootstrap complete.
I20250624 07:59:00.678493 22354 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Time spent bootstrapping tablet: real 0.171s user 0.136s sys 0.024s
I20250624 07:59:00.680619 22354 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:00.681306 22354 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Initialized, Role: FOLLOWER
I20250624 07:59:00.681923 22354 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:00.683172 22463 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
I20250624 07:59:00.683773 22354 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250624 07:59:00.906186 22467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:00.906900 22467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:00.907601 22467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:00.939780 22467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:00.940640 22467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:59:00.976164 22467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:42491
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=34685
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:00.977495 22467 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:00.979157 22467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:00.997285 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:01.011538 22306 debug-util.cc:398] Leaking SignalData structure 0x7b08000878e0 after lost signal to thread 22243
I20250624 07:59:02.076576 22484 raft_consensus.cc:491] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:02.077148 22484 raft_consensus.cc:513] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:02.080209 22484 leader_election.cc:290] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491), 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
W20250624 07:59:02.097600 22351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.219.194:42491: connect: Connection refused (error 111)
W20250624 07:59:02.104307 22351 leader_election.cc:336] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491): Network error: Client connection negotiation failed: client connection to 127.17.219.194:42491: connect: Connection refused (error 111)
W20250624 07:59:02.105126 22349 leader_election.cc:336] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
I20250624 07:59:02.105571 22349 leader_election.cc:304] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318; no voters: 9086f0567846438fad679c6b89da2f70, 9abb301874ab40d1b459bd027b1ec047
I20250624 07:59:02.106491 22484 raft_consensus.cc:2747] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250624 07:59:00.997502 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:00.999569 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:02.261772 22480 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:59:02.261813 22467 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:02.267060 22467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:02.269807 22467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:02.271315 22467 hybrid_clock.cc:648] HybridClock initialized: now 1750751942271281 us; error 49 us; skew 500 ppm
I20250624 07:59:02.272118 22467 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:02.279412 22467 webserver.cc:469] Webserver started at http://127.17.219.194:34685/ using document root <none> and password file <none>
I20250624 07:59:02.280329 22467 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:02.280553 22467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:02.288729 22467 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250624 07:59:02.293740 22492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:02.295111 22467 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:59:02.295579 22467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9086f0567846438fad679c6b89da2f70"
format_stamp: "Formatted at 2025-06-24 07:58:32 on dist-test-slave-qhgl"
I20250624 07:59:02.298473 22467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:02.349777 22484 raft_consensus.cc:491] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:02.350262 22484 raft_consensus.cc:513] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:02.352010 22484 leader_election.cc:290] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491), 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
W20250624 07:59:02.356956 22349 leader_election.cc:336] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
W20250624 07:59:02.358124 22351 leader_election.cc:336] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491): Network error: Client connection negotiation failed: client connection to 127.17.219.194:42491: connect: Connection refused (error 111)
I20250624 07:59:02.358539 22351 leader_election.cc:304] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318; no voters: 9086f0567846438fad679c6b89da2f70, 9abb301874ab40d1b459bd027b1ec047
I20250624 07:59:02.359266 22484 raft_consensus.cc:2747] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250624 07:59:02.359494 22467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:02.361011 22467 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:02.361433 22467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:02.364176 22467 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:02.370822 22500 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250624 07:59:02.390309 22467 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250624 07:59:02.390580 22467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.002s sys 0.000s
I20250624 07:59:02.390875 22467 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250624 07:59:02.396241 22500 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Bootstrap starting.
I20250624 07:59:02.399194 22467 ts_tablet_manager.cc:610] Registered 2 tablets
I20250624 07:59:02.399394 22467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.006s sys 0.002s
I20250624 07:59:02.471247 22500 log.cc:826] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:02.605461 22500 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:02.606691 22500 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Bootstrap complete.
I20250624 07:59:02.608999 22500 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Time spent bootstrapping tablet: real 0.213s user 0.158s sys 0.048s
I20250624 07:59:02.638962 22467 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:42491
I20250624 07:59:02.639120 22607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:42491 every 8 connection(s)
I20250624 07:59:02.636994 22500 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:02.640527 22500 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Initialized, Role: FOLLOWER
I20250624 07:59:02.641484 22500 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:02.642628 22467 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:59:02.645184 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 22467
I20250624 07:59:02.646347 22500 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Time spent starting tablet: real 0.037s user 0.032s sys 0.002s
I20250624 07:59:02.647480 22500 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap starting.
I20250624 07:59:02.647541 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:35163
--local_ip_for_outbound_sockets=127.17.219.195
--tserver_master_addrs=127.17.219.254:35921
--webserver_port=37537
--webserver_interface=127.17.219.195
--builtin_ntp_servers=127.17.219.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 07:59:02.709813 22609 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:59:02.710371 22609 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:02.711690 22609 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:59:02.716645 22275 ts_manager.cc:194] Registered new tserver with Master: 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491)
I20250624 07:59:02.721431 22275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:47613
I20250624 07:59:02.824060 22500 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:02.824996 22500 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Bootstrap complete.
I20250624 07:59:02.826581 22500 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Time spent bootstrapping tablet: real 0.179s user 0.146s sys 0.024s
I20250624 07:59:02.829012 22500 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:02.829711 22500 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Initialized, Role: FOLLOWER
I20250624 07:59:02.830341 22500 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:02.831681 22609 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
I20250624 07:59:02.832326 22500 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250624 07:59:03.026589 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:03.027138 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:03.027628 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:03.058764 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:03.059623 22610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:59:03.094424 22610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:35163
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=37537
--tserver_master_addrs=127.17.219.254:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:03.095849 22610 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:03.097420 22610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:03.112726 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:04.032845 22626 raft_consensus.cc:491] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:04.033480 22626 raft_consensus.cc:513] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.035311 22626 leader_election.cc:290] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491), 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
W20250624 07:59:04.053267 22349 leader_election.cc:336] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
I20250624 07:59:04.060577 22563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ce0df4c62349adb0ca6fe033243cb5" candidate_uuid: "73f2526ee2e54386be70659b58fe5318" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "9086f0567846438fad679c6b89da2f70" is_pre_election: true
I20250624 07:59:04.061374 22563 raft_consensus.cc:2466] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73f2526ee2e54386be70659b58fe5318 in term 2.
I20250624 07:59:04.062894 22351 leader_election.cc:304] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318, 9086f0567846438fad679c6b89da2f70; no voters: 9abb301874ab40d1b459bd027b1ec047
I20250624 07:59:04.063751 22626 raft_consensus.cc:2802] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250624 07:59:04.064426 22626 raft_consensus.cc:491] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:59:04.064960 22626 raft_consensus.cc:3058] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:59:04.079260 22626 raft_consensus.cc:513] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.082172 22626 leader_election.cc:290] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 election: Requested vote from peers 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491), 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
I20250624 07:59:04.083187 22563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ce0df4c62349adb0ca6fe033243cb5" candidate_uuid: "73f2526ee2e54386be70659b58fe5318" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "9086f0567846438fad679c6b89da2f70"
I20250624 07:59:04.083875 22563 raft_consensus.cc:3058] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Advancing to term 3
W20250624 07:59:04.087966 22349 leader_election.cc:336] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
I20250624 07:59:04.092811 22563 raft_consensus.cc:2466] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73f2526ee2e54386be70659b58fe5318 in term 3.
I20250624 07:59:04.093961 22351 leader_election.cc:304] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318, 9086f0567846438fad679c6b89da2f70; no voters: 9abb301874ab40d1b459bd027b1ec047
I20250624 07:59:04.094766 22626 raft_consensus.cc:2802] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 07:59:04.097378 22626 raft_consensus.cc:695] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 3 LEADER]: Becoming Leader. State: Replica: 73f2526ee2e54386be70659b58fe5318, State: Running, Role: LEADER
I20250624 07:59:04.098901 22626 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.113997 22275 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: term changed from 2 to 3, leader changed from <none> to 73f2526ee2e54386be70659b58fe5318 (127.17.219.193). New cstate: current_term: 3 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250624 07:59:03.113191 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:03.114095 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:04.307215 22622 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:59:04.307257 22610 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:04.312515 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:04.315433 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:04.316874 22610 hybrid_clock.cc:648] HybridClock initialized: now 1750751944316829 us; error 76 us; skew 500 ppm
I20250624 07:59:04.317667 22610 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:04.329103 22610 webserver.cc:469] Webserver started at http://127.17.219.195:37537/ using document root <none> and password file <none>
I20250624 07:59:04.330010 22610 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:04.330245 22610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:04.338843 22610 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250624 07:59:04.344343 22641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:04.345579 22610 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250624 07:59:04.346015 22610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9abb301874ab40d1b459bd027b1ec047"
format_stamp: "Formatted at 2025-06-24 07:58:34 on dist-test-slave-qhgl"
I20250624 07:59:04.348112 22610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:04.404625 22610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:04.406330 22610 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:04.406801 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:04.409525 22610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:04.415726 22648 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250624 07:59:04.431802 22610 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250624 07:59:04.432188 22610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.002s
I20250624 07:59:04.432497 22610 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250624 07:59:04.432456 22649 raft_consensus.cc:491] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:04.432917 22649 raft_consensus.cc:513] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.435277 22649 leader_election.cc:290] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225), 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
W20250624 07:59:04.441749 22495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
I20250624 07:59:04.443475 22648 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:59:04.454171 22610 ts_tablet_manager.cc:610] Registered 3 tablets
I20250624 07:59:04.455377 22610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.023s user 0.016s sys 0.005s
W20250624 07:59:04.458040 22495 leader_election.cc:336] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
I20250624 07:59:04.458196 22417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b87fe1dc267048d2b5403fe74189e17a" candidate_uuid: "9086f0567846438fad679c6b89da2f70" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "73f2526ee2e54386be70659b58fe5318" is_pre_election: true
I20250624 07:59:04.459101 22417 raft_consensus.cc:2466] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9086f0567846438fad679c6b89da2f70 in term 2.
I20250624 07:59:04.461113 22496 leader_election.cc:304] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318, 9086f0567846438fad679c6b89da2f70; no voters: 9abb301874ab40d1b459bd027b1ec047
I20250624 07:59:04.461851 22649 raft_consensus.cc:2802] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250624 07:59:04.462236 22649 raft_consensus.cc:491] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:59:04.462500 22649 raft_consensus.cc:3058] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:59:04.467435 22649 raft_consensus.cc:513] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.469075 22649 leader_election.cc:290] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 3 election: Requested vote from peers 73f2526ee2e54386be70659b58fe5318 (127.17.219.193:41225), 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
I20250624 07:59:04.470041 22417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b87fe1dc267048d2b5403fe74189e17a" candidate_uuid: "9086f0567846438fad679c6b89da2f70" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "73f2526ee2e54386be70659b58fe5318"
I20250624 07:59:04.470657 22417 raft_consensus.cc:3058] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 2 FOLLOWER]: Advancing to term 3
W20250624 07:59:04.475770 22495 leader_election.cc:336] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111)
I20250624 07:59:04.478680 22417 raft_consensus.cc:2466] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9086f0567846438fad679c6b89da2f70 in term 3.
I20250624 07:59:04.479794 22496 leader_election.cc:304] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73f2526ee2e54386be70659b58fe5318, 9086f0567846438fad679c6b89da2f70; no voters: 9abb301874ab40d1b459bd027b1ec047
I20250624 07:59:04.480647 22649 raft_consensus.cc:2802] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 07:59:04.482883 22649 raft_consensus.cc:695] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [term 3 LEADER]: Becoming Leader. State: Replica: 9086f0567846438fad679c6b89da2f70, State: Running, Role: LEADER
I20250624 07:59:04.483965 22649 consensus_queue.cc:237] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.503178 22275 catalog_manager.cc:5582] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "9086f0567846438fad679c6b89da2f70" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250624 07:59:04.524493 22648 log.cc:826] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Log is configured to *not* fsync() on all Append() calls
W20250624 07:59:04.555796 22349 consensus_peers.cc:487] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 -> Peer 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): Couldn't send request to peer 9abb301874ab40d1b459bd027b1ec047. Status: Network error: Client connection negotiation failed: client connection to 127.17.219.195:35163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:59:04.630889 22648 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:04.631805 22648 tablet_bootstrap.cc:492] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap complete.
I20250624 07:59:04.633231 22648 ts_tablet_manager.cc:1397] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.190s user 0.136s sys 0.042s
I20250624 07:59:04.656581 22648 raft_consensus.cc:357] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.659824 22648 raft_consensus.cc:738] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: FOLLOWER
I20250624 07:59:04.660974 22648 consensus_queue.cc:260] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.671393 22648 ts_tablet_manager.cc:1428] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.038s user 0.027s sys 0.009s
I20250624 07:59:04.672269 22648 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:59:04.692085 22610 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:35163
I20250624 07:59:04.692329 22763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:35163 every 8 connection(s)
I20250624 07:59:04.696167 22610 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:59:04.712651 22563 raft_consensus.cc:1273] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Refusing update from remote peer 73f2526ee2e54386be70659b58fe5318: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250624 07:59:04.714661 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 22610
I20250624 07:59:04.718775 22626 consensus_queue.cc:1035] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250624 07:59:04.740908 22764 heartbeater.cc:344] Connected to a master server at 127.17.219.254:35921
I20250624 07:59:04.741353 22764 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:04.742673 22764 heartbeater.cc:507] Master 127.17.219.254:35921 requested a full tablet report, sending...
I20250624 07:59:04.749094 22275 ts_manager.cc:194] Registered new tserver with Master: 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163)
I20250624 07:59:04.759200 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:59:04.759303 22275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:33723
I20250624 07:59:04.768527 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:59:04.773352 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73f2526ee2e54386be70659b58fe5318 to finish bootstrapping
I20250624 07:59:04.792466 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9086f0567846438fad679c6b89da2f70 to finish bootstrapping
I20250624 07:59:04.800556 22417 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } }
I20250624 07:59:04.805047 22563 raft_consensus.cc:1273] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Refusing update from remote peer 73f2526ee2e54386be70659b58fe5318: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250624 07:59:04.806604 22772 consensus_queue.cc:1035] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250624 07:59:04.809052 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9abb301874ab40d1b459bd027b1ec047 to finish bootstrapping
I20250624 07:59:04.812729 22775 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } }
I20250624 07:59:04.814280 22563 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } }
I20250624 07:59:04.823221 22261 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 01ce0df4c62349adb0ca6fe033243cb5 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 07:59:04.827266 22274 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 reported cstate change: config changed from index 11 to 13, VOTER 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195) evicted. New cstate: current_term: 3 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } attrs { promote: false } } }
I20250624 07:59:04.838274 22417 consensus_queue.cc:237] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } }
I20250624 07:59:04.840022 22775 raft_consensus.cc:2953] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 9086f0567846438fad679c6b89da2f70 (127.17.219.194) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } } }
I20250624 07:59:04.842902 22648 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:04.843766 22648 tablet_bootstrap.cc:492] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Bootstrap complete.
I20250624 07:59:04.845279 22648 ts_tablet_manager.cc:1397] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.173s user 0.143s sys 0.018s
I20250624 07:59:04.846071 22261 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 01ce0df4c62349adb0ca6fe033243cb5 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 07:59:04.847625 22648 raft_consensus.cc:357] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.848388 22648 raft_consensus.cc:738] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: FOLLOWER
I20250624 07:59:04.848959 22648 consensus_queue.cc:260] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9086f0567846438fad679c6b89da2f70" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 42491 } } peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false } } peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false } }
I20250624 07:59:04.849817 22275 catalog_manager.cc:5582] T 01ce0df4c62349adb0ca6fe033243cb5 P 73f2526ee2e54386be70659b58fe5318 reported cstate change: config changed from index 13 to 14, VOTER 9086f0567846438fad679c6b89da2f70 (127.17.219.194) evicted. New cstate: current_term: 3 leader_uuid: "73f2526ee2e54386be70659b58fe5318" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } health_report { overall_health: HEALTHY } } }
I20250624 07:59:04.851024 22648 ts_tablet_manager.cc:1428] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250624 07:59:04.851912 22648 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap starting.
I20250624 07:59:04.854144 22764 heartbeater.cc:499] Master 127.17.219.254:35921 was elected leader, sending a full tablet report...
I20250624 07:59:04.863790 22698 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ce0df4c62349adb0ca6fe033243cb5 with delete_type TABLET_DATA_TOMBSTONED (TS 9abb301874ab40d1b459bd027b1ec047 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:47626
I20250624 07:59:04.875928 22543 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ce0df4c62349adb0ca6fe033243cb5 with delete_type TABLET_DATA_TOMBSTONED (TS 9086f0567846438fad679c6b89da2f70 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:60804
I20250624 07:59:04.879369 22782 tablet_replica.cc:331] stopping tablet replica
I20250624 07:59:04.880010 22782 raft_consensus.cc:2241] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250624 07:59:04.880359 22781 tablet_replica.cc:331] stopping tablet replica
I20250624 07:59:04.880492 22782 raft_consensus.cc:2270] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250624 07:59:04.881569 22781 raft_consensus.cc:2241] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250624 07:59:04.882334 22781 raft_consensus.cc:2270] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 07:59:04.883090 22782 ts_tablet_manager.cc:1905] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 07:59:04.886204 22781 ts_tablet_manager.cc:1905] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 07:59:04.894874 22782 ts_tablet_manager.cc:1918] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250624 07:59:04.895206 22782 log.cc:1199] T 01ce0df4c62349adb0ca6fe033243cb5 P 9086f0567846438fad679c6b89da2f70: Deleting WAL directory at /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/wals/01ce0df4c62349adb0ca6fe033243cb5
I20250624 07:59:04.896605 22262 catalog_manager.cc:4928] TS 9086f0567846438fad679c6b89da2f70 (127.17.219.194:42491): tablet 01ce0df4c62349adb0ca6fe033243cb5 (table TestTable [id=29fa83bc689b4c45b26be3f7ac0f727c]) successfully deleted
I20250624 07:59:04.904351 22781 ts_tablet_manager.cc:1918] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250624 07:59:04.904771 22781 log.cc:1199] T 01ce0df4c62349adb0ca6fe033243cb5 P 9abb301874ab40d1b459bd027b1ec047: Deleting WAL directory at /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/wals/01ce0df4c62349adb0ca6fe033243cb5
I20250624 07:59:04.906517 22260 catalog_manager.cc:4928] TS 9abb301874ab40d1b459bd027b1ec047 (127.17.219.195:35163): tablet 01ce0df4c62349adb0ca6fe033243cb5 (table TestTable [id=29fa83bc689b4c45b26be3f7ac0f727c]) successfully deleted
I20250624 07:59:04.927949 22417 raft_consensus.cc:1273] T b87fe1dc267048d2b5403fe74189e17a P 73f2526ee2e54386be70659b58fe5318 [term 3 FOLLOWER]: Refusing update from remote peer 9086f0567846438fad679c6b89da2f70: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250624 07:59:04.929322 22649 consensus_queue.cc:1035] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f2526ee2e54386be70659b58fe5318" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 41225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250624 07:59:04.951109 22718 raft_consensus.cc:3058] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:59:04.956713 22718 raft_consensus.cc:1273] T b87fe1dc267048d2b5403fe74189e17a P 9abb301874ab40d1b459bd027b1ec047 [term 3 FOLLOWER]: Refusing update from remote peer 9086f0567846438fad679c6b89da2f70: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250624 07:59:04.958248 22649 consensus_queue.cc:1035] T b87fe1dc267048d2b5403fe74189e17a P 9086f0567846438fad679c6b89da2f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250624 07:59:04.981348 22648 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:59:04.982041 22648 tablet_bootstrap.cc:492] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Bootstrap complete.
I20250624 07:59:04.983326 22648 ts_tablet_manager.cc:1397] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Time spent bootstrapping tablet: real 0.132s user 0.111s sys 0.015s
I20250624 07:59:04.984871 22648 raft_consensus.cc:357] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:59:04.985208 22648 raft_consensus.cc:738] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Initialized, Role: FOLLOWER
I20250624 07:59:04.985683 22648 consensus_queue.cc:260] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:59:04.986037 22648 raft_consensus.cc:397] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:59:04.986234 22648 raft_consensus.cc:491] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:59:04.986466 22648 raft_consensus.cc:3058] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:59:04.990032 22648 raft_consensus.cc:513] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:59:04.990645 22648 leader_election.cc:304] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abb301874ab40d1b459bd027b1ec047; no voters:
I20250624 07:59:04.991084 22648 leader_election.cc:290] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250624 07:59:04.991326 22748 raft_consensus.cc:2802] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 07:59:04.993463 22748 raft_consensus.cc:695] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [term 3 LEADER]: Becoming Leader. State: Replica: 9abb301874ab40d1b459bd027b1ec047, State: Running, Role: LEADER
I20250624 07:59:04.994205 22648 ts_tablet_manager.cc:1428] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250624 07:59:04.994336 22748 consensus_queue.cc:237] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } }
I20250624 07:59:05.000742 22275 catalog_manager.cc:5582] T a56578d5b15841ffadd4b990951c28e1 P 9abb301874ab40d1b459bd027b1ec047 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "9abb301874ab40d1b459bd027b1ec047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb301874ab40d1b459bd027b1ec047" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 35163 } health_report { overall_health: HEALTHY } } }
W20250624 07:59:14.578840 22306 debug-util.cc:398] Leaking SignalData structure 0x7b08000870e0 after lost signal to thread 22243
W20250624 07:59:14.579525 22306 debug-util.cc:398] Leaking SignalData structure 0x7b080006f460 after lost signal to thread 22309
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Timed out: status_and_schema {
tablet_status {
tablet_id: "a56578d5b15841ffadd4b990951c28e1"
table_name: "TestTable2"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415715
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: LEADER
}
status_and_schema {
tablet_status {
tablet_id: "b87fe1dc267048d2b5403fe74189e17a"
table_name: "TestTable1"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415801
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: FOLLOWER
}
status_and_schema {
tablet_status {
tablet_id: "01ce0df4c62349adb0ca6fe033243cb5"
table_name: "TestTable"
state: STOPPED
last_status: "Deleted tablet blocks from disk"
estimated_on_disk_size: 8401345
tablet_data_state: TABLET_DATA_TOMBSTONED
partition {
partition_key_start: ""
partition_key_end: ""
}
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: UNKNOWN_ROLE
}
I20250624 07:59:24.834012 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 22312
I20250624 07:59:24.862443 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 22467
I20250624 07:59:24.892606 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 22610
I20250624 07:59:24.921926 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 22242
2025-06-24T07:59:24Z chronyd exiting
I20250624 07:59:24.975749 18287 test_util.cc:183] -----------------------------------------------
I20250624 07:59:24.975953 18287 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750751842019710-18287-0
[ FAILED ] AdminCliTest.TestRebuildTables (58466 ms)
[----------] 5 tests from AdminCliTest (122888 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250624 07:59:24.979485 18287 test_util.cc:276] Using random seed: -1258921726
I20250624 07:59:24.983949 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:59:24.984115 18287 ts_itest-base.cc:116] --------------
I20250624 07:59:24.984234 18287 ts_itest-base.cc:117] 5 tablet servers
I20250624 07:59:24.984333 18287 ts_itest-base.cc:118] 3 replicas per TS
I20250624 07:59:24.984450 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:59:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:59:25Z Disabled control of system clock
I20250624 07:59:25.029126 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:40463
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:40463
--raft_prepare_replacement_before_eviction=true with env {}
W20250624 07:59:25.336547 22812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:25.337200 22812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:25.337682 22812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:25.369117 22812 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 07:59:25.369562 22812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:59:25.369830 22812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:25.370081 22812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:59:25.370332 22812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:59:25.405840 22812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:40463
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:25.407218 22812 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:25.408852 22812 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:25.425001 22818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:25.425100 22819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:25.425001 22821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:25.426594 22812 server_base.cc:1048] running on GCE node
I20250624 07:59:26.622043 22812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:26.625272 22812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:26.626794 22812 hybrid_clock.cc:648] HybridClock initialized: now 1750751966626735 us; error 75 us; skew 500 ppm
I20250624 07:59:26.627614 22812 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:26.641418 22812 webserver.cc:469] Webserver started at http://127.17.219.254:35719/ using document root <none> and password file <none>
I20250624 07:59:26.642416 22812 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:26.642618 22812 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:26.643108 22812 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:26.649284 22812 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5685bfb21b764273b3bdc2fb47ee89d2"
format_stamp: "Formatted at 2025-06-24 07:59:26 on dist-test-slave-qhgl"
I20250624 07:59:26.650425 22812 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5685bfb21b764273b3bdc2fb47ee89d2"
format_stamp: "Formatted at 2025-06-24 07:59:26 on dist-test-slave-qhgl"
I20250624 07:59:26.658298 22812 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 07:59:26.664208 22828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:26.665560 22812 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250624 07:59:26.665927 22812 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5685bfb21b764273b3bdc2fb47ee89d2"
format_stamp: "Formatted at 2025-06-24 07:59:26 on dist-test-slave-qhgl"
I20250624 07:59:26.666301 22812 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:26.723951 22812 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:26.725482 22812 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:26.725934 22812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:26.801155 22812 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:40463
I20250624 07:59:26.801306 22879 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:40463 every 8 connection(s)
I20250624 07:59:26.803918 22812 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:59:26.809548 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 22812
I20250624 07:59:26.809676 22880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:26.809991 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:59:26.836297 22880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2: Bootstrap starting.
I20250624 07:59:26.842566 22880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:26.844517 22880 log.cc:826] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:26.849674 22880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2: No bootstrap required, opened a new log
I20250624 07:59:26.870819 22880 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40463 } }
I20250624 07:59:26.871807 22880 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:26.872032 22880 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5685bfb21b764273b3bdc2fb47ee89d2, State: Initialized, Role: FOLLOWER
I20250624 07:59:26.872915 22880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40463 } }
I20250624 07:59:26.873524 22880 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:59:26.873838 22880 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:59:26.874159 22880 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:26.878763 22880 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40463 } }
I20250624 07:59:26.879575 22880 leader_election.cc:304] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5685bfb21b764273b3bdc2fb47ee89d2; no voters:
I20250624 07:59:26.881886 22880 leader_election.cc:290] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:59:26.882663 22885 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:59:26.885115 22885 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5685bfb21b764273b3bdc2fb47ee89d2, State: Running, Role: LEADER
I20250624 07:59:26.885902 22885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40463 } }
I20250624 07:59:26.886420 22880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:59:26.896919 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40463 } } }
I20250624 07:59:26.897315 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5685bfb21b764273b3bdc2fb47ee89d2. Latest consensus state: current_term: 1 leader_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685bfb21b764273b3bdc2fb47ee89d2" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 40463 } } }
I20250624 07:59:26.898047 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [sys.catalog]: This master's current role is: LEADER
I20250624 07:59:26.898694 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2 [sys.catalog]: This master's current role is: LEADER
I20250624 07:59:26.903729 22893 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:59:26.914600 22893 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:59:26.932943 22893 catalog_manager.cc:1349] Generated new cluster ID: e2f90e2999864fe1903f53be5c0f45ce
I20250624 07:59:26.933328 22893 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:59:26.965667 22893 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:59:26.967317 22893 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:59:26.984081 22893 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5685bfb21b764273b3bdc2fb47ee89d2: Generated new TSK 0
I20250624 07:59:26.985330 22893 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:59:27.008724 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--builtin_ntp_servers=127.17.219.212:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250624 07:59:27.328945 22904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:27.329488 22904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:27.329985 22904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:27.363312 22904 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 07:59:27.363719 22904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:27.364512 22904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:59:27.400457 22904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:27.401846 22904 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:27.403590 22904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:27.425069 22910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:28.818571 22909 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 22904
W20250624 07:59:27.425163 22911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:28.935847 22904 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.519s user 0.451s sys 0.921s
W20250624 07:59:28.937763 22904 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.521s user 0.451s sys 0.922s
W20250624 07:59:28.938172 22912 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1517 milliseconds
I20250624 07:59:28.938319 22904 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250624 07:59:28.938381 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:28.942430 22904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:28.945520 22904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:28.947021 22904 hybrid_clock.cc:648] HybridClock initialized: now 1750751968946953 us; error 83 us; skew 500 ppm
I20250624 07:59:28.947897 22904 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:28.955368 22904 webserver.cc:469] Webserver started at http://127.17.219.193:46869/ using document root <none> and password file <none>
I20250624 07:59:28.956357 22904 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:28.956561 22904 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:28.957028 22904 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:28.961611 22904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d8560962e0d04bd7974d7b4098b90747"
format_stamp: "Formatted at 2025-06-24 07:59:28 on dist-test-slave-qhgl"
I20250624 07:59:28.962818 22904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d8560962e0d04bd7974d7b4098b90747"
format_stamp: "Formatted at 2025-06-24 07:59:28 on dist-test-slave-qhgl"
I20250624 07:59:28.970806 22904 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 07:59:28.977233 22920 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:28.978432 22904 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250624 07:59:28.978828 22904 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d8560962e0d04bd7974d7b4098b90747"
format_stamp: "Formatted at 2025-06-24 07:59:28 on dist-test-slave-qhgl"
I20250624 07:59:28.979182 22904 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:29.038180 22904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:29.039748 22904 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:29.040311 22904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:29.043045 22904 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:29.047789 22904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:29.048025 22904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:29.048283 22904 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:29.048452 22904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:29.222167 22904 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:36273
I20250624 07:59:29.222280 23032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:36273 every 8 connection(s)
I20250624 07:59:29.224865 22904 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:59:29.234937 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 22904
I20250624 07:59:29.235451 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:59:29.243171 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--builtin_ntp_servers=127.17.219.212:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 07:59:29.247539 23033 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40463
I20250624 07:59:29.247992 23033 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:29.249042 23033 heartbeater.cc:507] Master 127.17.219.254:40463 requested a full tablet report, sending...
I20250624 07:59:29.251636 22845 ts_manager.cc:194] Registered new tserver with Master: d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273)
I20250624 07:59:29.254113 22845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:60627
W20250624 07:59:29.594080 23037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:29.594621 23037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:29.595217 23037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:29.627686 23037 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 07:59:29.628209 23037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:29.629199 23037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:59:29.664355 23037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:29.665689 23037 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:29.667392 23037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:29.684603 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:30.259235 23033 heartbeater.cc:499] Master 127.17.219.254:40463 was elected leader, sending a full tablet report...
W20250624 07:59:29.684687 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:29.685494 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:29.687077 23037 server_base.cc:1048] running on GCE node
I20250624 07:59:30.860237 23037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:30.862458 23037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:30.863824 23037 hybrid_clock.cc:648] HybridClock initialized: now 1750751970863787 us; error 52 us; skew 500 ppm
I20250624 07:59:30.864650 23037 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:30.870967 23037 webserver.cc:469] Webserver started at http://127.17.219.194:35273/ using document root <none> and password file <none>
I20250624 07:59:30.871867 23037 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:30.872087 23037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:30.872531 23037 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:30.877007 23037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "beb87c1894fa4e84bc28d255dca150f0"
format_stamp: "Formatted at 2025-06-24 07:59:30 on dist-test-slave-qhgl"
I20250624 07:59:30.878098 23037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "beb87c1894fa4e84bc28d255dca150f0"
format_stamp: "Formatted at 2025-06-24 07:59:30 on dist-test-slave-qhgl"
I20250624 07:59:30.885473 23037 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250624 07:59:30.891363 23053 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:30.892463 23037 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250624 07:59:30.892815 23037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "beb87c1894fa4e84bc28d255dca150f0"
format_stamp: "Formatted at 2025-06-24 07:59:30 on dist-test-slave-qhgl"
I20250624 07:59:30.893173 23037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:30.976306 23037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:30.977730 23037 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:30.978173 23037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:30.980839 23037 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:30.985186 23037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:30.985410 23037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:30.985661 23037 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:30.985818 23037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:31.131728 23037 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:44777
I20250624 07:59:31.131879 23165 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:44777 every 8 connection(s)
I20250624 07:59:31.134367 23037 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 07:59:31.139902 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23037
I20250624 07:59:31.140439 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 07:59:31.147338 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--builtin_ntp_servers=127.17.219.212:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 07:59:31.158414 23166 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40463
I20250624 07:59:31.159000 23166 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:31.160151 23166 heartbeater.cc:507] Master 127.17.219.254:40463 requested a full tablet report, sending...
I20250624 07:59:31.162350 22845 ts_manager.cc:194] Registered new tserver with Master: beb87c1894fa4e84bc28d255dca150f0 (127.17.219.194:44777)
I20250624 07:59:31.163620 22845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:57769
W20250624 07:59:31.481146 23170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:31.481700 23170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:31.482228 23170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:31.515386 23170 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 07:59:31.515838 23170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:31.516669 23170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:59:31.553994 23170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:31.555429 23170 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:31.557176 23170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:31.574853 23177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:32.166913 23166 heartbeater.cc:499] Master 127.17.219.254:40463 was elected leader, sending a full tablet report...
W20250624 07:59:31.574942 23176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:31.574980 23179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:32.761778 23178 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 07:59:32.761866 23170 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:32.766593 23170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:32.769323 23170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:32.770795 23170 hybrid_clock.cc:648] HybridClock initialized: now 1750751972770743 us; error 62 us; skew 500 ppm
I20250624 07:59:32.771593 23170 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:32.781513 23170 webserver.cc:469] Webserver started at http://127.17.219.195:40119/ using document root <none> and password file <none>
I20250624 07:59:32.782418 23170 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:32.782605 23170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:32.783079 23170 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:32.787526 23170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d166c96f1d4e48e0ab5fcf1a36de2123"
format_stamp: "Formatted at 2025-06-24 07:59:32 on dist-test-slave-qhgl"
I20250624 07:59:32.788678 23170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d166c96f1d4e48e0ab5fcf1a36de2123"
format_stamp: "Formatted at 2025-06-24 07:59:32 on dist-test-slave-qhgl"
I20250624 07:59:32.795583 23170 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250624 07:59:32.801692 23187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:32.802845 23170 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.001s
I20250624 07:59:32.803154 23170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d166c96f1d4e48e0ab5fcf1a36de2123"
format_stamp: "Formatted at 2025-06-24 07:59:32 on dist-test-slave-qhgl"
I20250624 07:59:32.803470 23170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:32.866822 23170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:32.868263 23170 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:32.868680 23170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:32.871292 23170 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:32.875312 23170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:32.875506 23170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:32.875785 23170 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:32.875929 23170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:33.011435 23170 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:41249
I20250624 07:59:33.011533 23299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:41249 every 8 connection(s)
I20250624 07:59:33.013999 23170 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 07:59:33.022923 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23170
I20250624 07:59:33.023497 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 07:59:33.030187 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.196:0
--local_ip_for_outbound_sockets=127.17.219.196
--webserver_interface=127.17.219.196
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--builtin_ntp_servers=127.17.219.212:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 07:59:33.034186 23300 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40463
I20250624 07:59:33.034646 23300 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:33.035684 23300 heartbeater.cc:507] Master 127.17.219.254:40463 requested a full tablet report, sending...
I20250624 07:59:33.037721 22845 ts_manager.cc:194] Registered new tserver with Master: d166c96f1d4e48e0ab5fcf1a36de2123 (127.17.219.195:41249)
I20250624 07:59:33.039134 22845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:60909
W20250624 07:59:33.352038 23304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:33.352486 23304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:33.352931 23304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:33.384521 23304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 07:59:33.384917 23304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:33.385653 23304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.196
I20250624 07:59:33.419672 23304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.219.196
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.196
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:33.420965 23304 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:33.422550 23304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:33.437211 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:34.042842 23300 heartbeater.cc:499] Master 127.17.219.254:40463 was elected leader, sending a full tablet report...
W20250624 07:59:34.837659 23309 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 23304
W20250624 07:59:35.143924 23309 kernel_stack_watchdog.cc:198] Thread 23304 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250624 07:59:33.437238 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:35.144917 23304 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.711s user 0.605s sys 1.104s
W20250624 07:59:35.145259 23304 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.711s user 0.605s sys 1.104s
W20250624 07:59:35.145289 23312 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1708 milliseconds
W20250624 07:59:35.147074 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:35.150851 23304 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:35.151892 23304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:35.153817 23304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:35.155154 23304 hybrid_clock.cc:648] HybridClock initialized: now 1750751975155130 us; error 32 us; skew 500 ppm
I20250624 07:59:35.156154 23304 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:35.162097 23304 webserver.cc:469] Webserver started at http://127.17.219.196:37613/ using document root <none> and password file <none>
I20250624 07:59:35.163065 23304 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:35.163287 23304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:35.163734 23304 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:35.168116 23304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5de3646fc9734c8eb36b840427f5eb07"
format_stamp: "Formatted at 2025-06-24 07:59:35 on dist-test-slave-qhgl"
I20250624 07:59:35.169216 23304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5de3646fc9734c8eb36b840427f5eb07"
format_stamp: "Formatted at 2025-06-24 07:59:35 on dist-test-slave-qhgl"
I20250624 07:59:35.176121 23304 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250624 07:59:35.182468 23320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:35.183662 23304 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250624 07:59:35.183959 23304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5de3646fc9734c8eb36b840427f5eb07"
format_stamp: "Formatted at 2025-06-24 07:59:35 on dist-test-slave-qhgl"
I20250624 07:59:35.184366 23304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:35.237001 23304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:35.238382 23304 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:35.238802 23304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:35.241200 23304 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:35.245126 23304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:35.245313 23304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:35.245514 23304 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:35.245643 23304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:35.382352 23304 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.196:33685
I20250624 07:59:35.382475 23433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.196:33685 every 8 connection(s)
I20250624 07:59:35.384871 23304 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250624 07:59:35.392190 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23304
I20250624 07:59:35.392577 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250624 07:59:35.399691 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.197:0
--local_ip_for_outbound_sockets=127.17.219.197
--webserver_interface=127.17.219.197
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--builtin_ntp_servers=127.17.219.212:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 07:59:35.406769 23434 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40463
I20250624 07:59:35.407361 23434 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:35.408689 23434 heartbeater.cc:507] Master 127.17.219.254:40463 requested a full tablet report, sending...
I20250624 07:59:35.411106 22845 ts_manager.cc:194] Registered new tserver with Master: 5de3646fc9734c8eb36b840427f5eb07 (127.17.219.196:33685)
I20250624 07:59:35.412582 22845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.196:42009
W20250624 07:59:35.697667 23438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:35.698172 23438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:35.698669 23438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:35.730605 23438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 07:59:35.731046 23438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:35.731839 23438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.197
I20250624 07:59:35.766242 23438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.219.197
--webserver_port=0
--tserver_master_addrs=127.17.219.254:40463
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.197
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:35.767587 23438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:35.769155 23438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:35.784255 23444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:36.415819 23434 heartbeater.cc:499] Master 127.17.219.254:40463 was elected leader, sending a full tablet report...
W20250624 07:59:35.784916 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:35.785221 23445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:35.786875 23438 server_base.cc:1048] running on GCE node
I20250624 07:59:36.974551 23438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:36.976961 23438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:36.978343 23438 hybrid_clock.cc:648] HybridClock initialized: now 1750751976978302 us; error 74 us; skew 500 ppm
I20250624 07:59:36.979203 23438 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:36.992444 23438 webserver.cc:469] Webserver started at http://127.17.219.197:37945/ using document root <none> and password file <none>
I20250624 07:59:36.993994 23438 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:36.994333 23438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:36.995075 23438 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:37.003399 23438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "dd52b36c18e5403f9919f5e764c9c436"
format_stamp: "Formatted at 2025-06-24 07:59:36 on dist-test-slave-qhgl"
I20250624 07:59:37.005239 23438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "dd52b36c18e5403f9919f5e764c9c436"
format_stamp: "Formatted at 2025-06-24 07:59:36 on dist-test-slave-qhgl"
I20250624 07:59:37.015378 23438 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20250624 07:59:37.025348 23454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:37.027031 23438 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20250624 07:59:37.027491 23438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "dd52b36c18e5403f9919f5e764c9c436"
format_stamp: "Formatted at 2025-06-24 07:59:36 on dist-test-slave-qhgl"
I20250624 07:59:37.027942 23438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:37.078341 23438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:37.080087 23438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:37.080607 23438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:37.083432 23438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:37.088008 23438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:37.088294 23438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:37.088615 23438 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:37.088827 23438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:37.275507 23438 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.197:43169
I20250624 07:59:37.275636 23566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.197:43169 every 8 connection(s)
I20250624 07:59:37.278194 23438 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250624 07:59:37.284547 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23438
I20250624 07:59:37.285109 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250624 07:59:37.305073 23567 heartbeater.cc:344] Connected to a master server at 127.17.219.254:40463
I20250624 07:59:37.305629 23567 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:37.307304 23567 heartbeater.cc:507] Master 127.17.219.254:40463 requested a full tablet report, sending...
I20250624 07:59:37.310115 22844 ts_manager.cc:194] Registered new tserver with Master: dd52b36c18e5403f9919f5e764c9c436 (127.17.219.197:43169)
I20250624 07:59:37.312286 22844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.197:38975
I20250624 07:59:37.326736 18287 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250624 07:59:37.364929 22844 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41416:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 07:59:37.462132 23101 tablet_service.cc:1468] Processing CreateTablet for tablet ef624ecd97e249408d204524844bf523 (DEFAULT_TABLE table=TestTable [id=34a0a6005ad74616a87fc0df8831f57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:37.464437 23101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef624ecd97e249408d204524844bf523. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:37.469130 23368 tablet_service.cc:1468] Processing CreateTablet for tablet ef624ecd97e249408d204524844bf523 (DEFAULT_TABLE table=TestTable [id=34a0a6005ad74616a87fc0df8831f57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:37.470891 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef624ecd97e249408d204524844bf523. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:37.472204 22968 tablet_service.cc:1468] Processing CreateTablet for tablet ef624ecd97e249408d204524844bf523 (DEFAULT_TABLE table=TestTable [id=34a0a6005ad74616a87fc0df8831f57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:37.474887 22968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef624ecd97e249408d204524844bf523. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:37.498517 23587 tablet_bootstrap.cc:492] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: Bootstrap starting.
I20250624 07:59:37.501281 23588 tablet_bootstrap.cc:492] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: Bootstrap starting.
I20250624 07:59:37.507383 23587 tablet_bootstrap.cc:654] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:37.509713 23589 tablet_bootstrap.cc:492] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: Bootstrap starting.
I20250624 07:59:37.510811 23588 tablet_bootstrap.cc:654] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:37.513389 23587 log.cc:826] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:37.513597 23588 log.cc:826] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:37.519291 23589 tablet_bootstrap.cc:654] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:37.521132 23588 tablet_bootstrap.cc:492] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: No bootstrap required, opened a new log
I20250624 07:59:37.521473 23587 tablet_bootstrap.cc:492] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: No bootstrap required, opened a new log
I20250624 07:59:37.521675 23588 ts_tablet_manager.cc:1397] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.013s
I20250624 07:59:37.522037 23587 ts_tablet_manager.cc:1397] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20250624 07:59:37.522315 23589 log.cc:826] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:37.529196 23589 tablet_bootstrap.cc:492] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: No bootstrap required, opened a new log
I20250624 07:59:37.529884 23589 ts_tablet_manager.cc:1397] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20250624 07:59:37.550891 23589 raft_consensus.cc:357] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.551973 23589 raft_consensus.cc:383] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:37.551512 23587 raft_consensus.cc:357] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.551776 23588 raft_consensus.cc:357] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.552263 23589 raft_consensus.cc:738] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8560962e0d04bd7974d7b4098b90747, State: Initialized, Role: FOLLOWER
I20250624 07:59:37.552522 23587 raft_consensus.cc:383] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:37.552646 23588 raft_consensus.cc:383] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:37.552881 23587 raft_consensus.cc:738] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de3646fc9734c8eb36b840427f5eb07, State: Initialized, Role: FOLLOWER
I20250624 07:59:37.552953 23588 raft_consensus.cc:738] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb87c1894fa4e84bc28d255dca150f0, State: Initialized, Role: FOLLOWER
I20250624 07:59:37.553129 23589 consensus_queue.cc:260] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.553825 23588 consensus_queue.cc:260] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.553815 23587 consensus_queue.cc:260] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.559243 23588 ts_tablet_manager.cc:1428] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: Time spent starting tablet: real 0.037s user 0.020s sys 0.015s
I20250624 07:59:37.562422 23587 ts_tablet_manager.cc:1428] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: Time spent starting tablet: real 0.040s user 0.029s sys 0.008s
I20250624 07:59:37.565369 23589 ts_tablet_manager.cc:1428] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: Time spent starting tablet: real 0.035s user 0.019s sys 0.012s
W20250624 07:59:37.641896 23435 tablet.cc:2378] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:59:37.646838 23167 tablet.cc:2378] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:59:37.743062 23034 tablet.cc:2378] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:59:37.819520 23594 raft_consensus.cc:491] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:37.820237 23594 raft_consensus.cc:513] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.824591 23594 leader_election.cc:290] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beb87c1894fa4e84bc28d255dca150f0 (127.17.219.194:44777), 5de3646fc9734c8eb36b840427f5eb07 (127.17.219.196:33685)
I20250624 07:59:37.842469 23389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef624ecd97e249408d204524844bf523" candidate_uuid: "d8560962e0d04bd7974d7b4098b90747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de3646fc9734c8eb36b840427f5eb07" is_pre_election: true
I20250624 07:59:37.843400 23389 raft_consensus.cc:2466] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8560962e0d04bd7974d7b4098b90747 in term 0.
I20250624 07:59:37.843555 23121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef624ecd97e249408d204524844bf523" candidate_uuid: "d8560962e0d04bd7974d7b4098b90747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb87c1894fa4e84bc28d255dca150f0" is_pre_election: true
I20250624 07:59:37.844447 23121 raft_consensus.cc:2466] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8560962e0d04bd7974d7b4098b90747 in term 0.
I20250624 07:59:37.844982 22923 leader_election.cc:304] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de3646fc9734c8eb36b840427f5eb07, d8560962e0d04bd7974d7b4098b90747; no voters:
I20250624 07:59:37.845844 23594 raft_consensus.cc:2802] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:59:37.846866 23594 raft_consensus.cc:491] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:59:37.847215 23594 raft_consensus.cc:3058] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:37.853935 23594 raft_consensus.cc:513] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.855876 23594 leader_election.cc:290] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [CANDIDATE]: Term 1 election: Requested vote from peers beb87c1894fa4e84bc28d255dca150f0 (127.17.219.194:44777), 5de3646fc9734c8eb36b840427f5eb07 (127.17.219.196:33685)
I20250624 07:59:37.856714 23121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef624ecd97e249408d204524844bf523" candidate_uuid: "d8560962e0d04bd7974d7b4098b90747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb87c1894fa4e84bc28d255dca150f0"
I20250624 07:59:37.857161 23121 raft_consensus.cc:3058] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:37.857551 23389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef624ecd97e249408d204524844bf523" candidate_uuid: "d8560962e0d04bd7974d7b4098b90747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de3646fc9734c8eb36b840427f5eb07"
I20250624 07:59:37.858121 23389 raft_consensus.cc:3058] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:37.863683 23121 raft_consensus.cc:2466] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8560962e0d04bd7974d7b4098b90747 in term 1.
I20250624 07:59:37.864646 23389 raft_consensus.cc:2466] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8560962e0d04bd7974d7b4098b90747 in term 1.
I20250624 07:59:37.864744 22921 leader_election.cc:304] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beb87c1894fa4e84bc28d255dca150f0, d8560962e0d04bd7974d7b4098b90747; no voters:
I20250624 07:59:37.865559 23594 raft_consensus.cc:2802] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:59:37.867630 23594 raft_consensus.cc:695] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [term 1 LEADER]: Becoming Leader. State: Replica: d8560962e0d04bd7974d7b4098b90747, State: Running, Role: LEADER
I20250624 07:59:37.868559 23594 consensus_queue.cc:237] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:37.881289 22845 catalog_manager.cc:5582] T ef624ecd97e249408d204524844bf523 P d8560962e0d04bd7974d7b4098b90747 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8560962e0d04bd7974d7b4098b90747 (127.17.219.193). New cstate: current_term: 1 leader_uuid: "d8560962e0d04bd7974d7b4098b90747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } health_report { overall_health: UNKNOWN } } }
I20250624 07:59:37.925196 18287 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250624 07:59:37.928751 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d8560962e0d04bd7974d7b4098b90747 to finish bootstrapping
I20250624 07:59:37.946352 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver beb87c1894fa4e84bc28d255dca150f0 to finish bootstrapping
I20250624 07:59:37.960067 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5de3646fc9734c8eb36b840427f5eb07 to finish bootstrapping
I20250624 07:59:37.972000 18287 test_util.cc:276] Using random seed: -1245929218
I20250624 07:59:38.000967 18287 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250624 07:59:38.002036 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 22904
W20250624 07:59:38.040002 23605 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:59:38.016272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.219.193:36273 (local address 127.0.0.1:55346)
0624 07:59:38.017259 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0624 07:59:38.017311 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0624 07:59:38.017614 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0624 07:59:38.029735 (+ 12121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":108}
W20250624 07:59:38.041887 23601 meta_cache.cc:302] tablet ef624ecd97e249408d204524844bf523: replica d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273) has failed: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:59:38.060735 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.080252 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.101956 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.114393 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.141887 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.158910 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.195451 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.214923 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.252548 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.270910 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
I20250624 07:59:38.315779 23567 heartbeater.cc:499] Master 127.17.219.254:40463 was elected leader, sending a full tablet report...
W20250624 07:59:38.318611 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.342511 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.397716 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.426537 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.483291 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.512785 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.574806 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.610821 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.686882 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.722091 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.802412 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.846884 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.937775 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:38.984184 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.076759 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.124063 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.175874 23601 meta_cache.cc:302] tablet ef624ecd97e249408d204524844bf523: replica d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273) has failed: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111)
W20250624 07:59:39.222957 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.271945 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.376518 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.426285 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
W20250624 07:59:39.534804 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
I20250624 07:59:39.544212 23619 raft_consensus.cc:491] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:39.544716 23619 raft_consensus.cc:513] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:39.547686 23619 leader_election.cc:290] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers beb87c1894fa4e84bc28d255dca150f0 (127.17.219.194:44777), d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273)
W20250624 07:59:39.570860 23321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111)
W20250624 07:59:39.581274 23321 leader_election.cc:336] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273): Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111)
W20250624 07:59:39.587437 23348 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38142: Illegal state: replica 5de3646fc9734c8eb36b840427f5eb07 is not leader of this config: current role FOLLOWER
I20250624 07:59:39.601305 23121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef624ecd97e249408d204524844bf523" candidate_uuid: "5de3646fc9734c8eb36b840427f5eb07" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb87c1894fa4e84bc28d255dca150f0" is_pre_election: true
I20250624 07:59:39.602231 23121 raft_consensus.cc:2466] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de3646fc9734c8eb36b840427f5eb07 in term 1.
I20250624 07:59:39.603507 23321 leader_election.cc:304] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5de3646fc9734c8eb36b840427f5eb07, beb87c1894fa4e84bc28d255dca150f0; no voters: d8560962e0d04bd7974d7b4098b90747
I20250624 07:59:39.604359 23619 raft_consensus.cc:2802] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 07:59:39.604807 23619 raft_consensus.cc:491] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:59:39.605239 23619 raft_consensus.cc:3058] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:59:39.614805 23619 raft_consensus.cc:513] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:39.617311 23619 leader_election.cc:290] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [CANDIDATE]: Term 2 election: Requested vote from peers beb87c1894fa4e84bc28d255dca150f0 (127.17.219.194:44777), d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273)
I20250624 07:59:39.618050 23121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef624ecd97e249408d204524844bf523" candidate_uuid: "5de3646fc9734c8eb36b840427f5eb07" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb87c1894fa4e84bc28d255dca150f0"
I20250624 07:59:39.618551 23121 raft_consensus.cc:3058] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 1 FOLLOWER]: Advancing to term 2
W20250624 07:59:39.624189 23321 leader_election.cc:336] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273): Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111)
I20250624 07:59:39.626570 23121 raft_consensus.cc:2466] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de3646fc9734c8eb36b840427f5eb07 in term 2.
I20250624 07:59:39.627657 23321 leader_election.cc:304] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5de3646fc9734c8eb36b840427f5eb07, beb87c1894fa4e84bc28d255dca150f0; no voters: d8560962e0d04bd7974d7b4098b90747
I20250624 07:59:39.628432 23619 raft_consensus.cc:2802] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:59:39.630869 23619 raft_consensus.cc:695] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [term 2 LEADER]: Becoming Leader. State: Replica: 5de3646fc9734c8eb36b840427f5eb07, State: Running, Role: LEADER
I20250624 07:59:39.632947 23619 consensus_queue.cc:237] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } }
I20250624 07:59:39.648836 22844 catalog_manager.cc:5582] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 reported cstate change: term changed from 1 to 2, leader changed from d8560962e0d04bd7974d7b4098b90747 (127.17.219.193) to 5de3646fc9734c8eb36b840427f5eb07 (127.17.219.196). New cstate: current_term: 2 leader_uuid: "5de3646fc9734c8eb36b840427f5eb07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8560962e0d04bd7974d7b4098b90747" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 36273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de3646fc9734c8eb36b840427f5eb07" member_type: VOTER last_known_addr { host: "127.17.219.196" port: 33685 } health_report { overall_health: HEALTHY } } }
W20250624 07:59:39.707681 23081 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41590: Illegal state: replica beb87c1894fa4e84bc28d255dca150f0 is not leader of this config: current role FOLLOWER
I20250624 07:59:39.773557 23121 raft_consensus.cc:1273] T ef624ecd97e249408d204524844bf523 P beb87c1894fa4e84bc28d255dca150f0 [term 2 FOLLOWER]: Refusing update from remote peer 5de3646fc9734c8eb36b840427f5eb07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 07:59:39.775449 23619 consensus_queue.cc:1035] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beb87c1894fa4e84bc28d255dca150f0" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250624 07:59:39.776243 23321 consensus_peers.cc:487] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 -> Peer d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273): Couldn't send request to peer d8560962e0d04bd7974d7b4098b90747. Status: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:59:39.799393 23631 mvcc.cc:204] Tried to move back new op lower bound from 7171080109128671232 to 7171080108589412352. Current Snapshot: MvccSnapshot[applied={T|T < 7171080109128671232}]
I20250624 07:59:40.247810 23368 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 07:59:40.309242 23101 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 07:59:40.320986 23235 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 07:59:40.356752 23502 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250624 07:59:42.037626 23321 consensus_peers.cc:487] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 -> Peer d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273): Couldn't send request to peer d8560962e0d04bd7974d7b4098b90747. Status: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250624 07:59:42.110248 23368 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 07:59:42.127365 23235 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 07:59:42.135048 23101 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 07:59:42.178184 23502 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250624 07:59:44.421772 23321 consensus_peers.cc:487] T ef624ecd97e249408d204524844bf523 P 5de3646fc9734c8eb36b840427f5eb07 -> Peer d8560962e0d04bd7974d7b4098b90747 (127.17.219.193:36273): Couldn't send request to peer d8560962e0d04bd7974d7b4098b90747. Status: Network error: Client connection negotiation failed: client connection to 127.17.219.193:36273: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250624 07:59:44.586074 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23037
I20250624 07:59:44.623164 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23170
I20250624 07:59:44.649403 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23304
I20250624 07:59:44.688169 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23438
I20250624 07:59:44.713050 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 22812
2025-06-24T07:59:44Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19795 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19795 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250624 07:59:44.775163 18287 test_util.cc:276] Using random seed: -1239126048
I20250624 07:59:44.779366 18287 ts_itest-base.cc:115] Starting cluster with:
I20250624 07:59:44.779529 18287 ts_itest-base.cc:116] --------------
I20250624 07:59:44.779690 18287 ts_itest-base.cc:117] 1 tablet servers
I20250624 07:59:44.779826 18287 ts_itest-base.cc:118] 1 replicas per TS
I20250624 07:59:44.779970 18287 ts_itest-base.cc:119] --------------
2025-06-24T07:59:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:59:44Z Disabled control of system clock
I20250624 07:59:44.825259 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:45509
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:45509 with env {}
W20250624 07:59:45.135149 23714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:45.135741 23714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:45.136142 23714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:45.167870 23714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:59:45.168162 23714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:45.168368 23714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:59:45.168561 23714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:59:45.204807 23714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:45509
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:45509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:45.206084 23714 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:45.207706 23714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:45.221338 23721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:45.221879 23720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:45.223944 23714 server_base.cc:1048] running on GCE node
W20250624 07:59:45.223534 23723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:46.401108 23714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:46.404728 23714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:46.406179 23714 hybrid_clock.cc:648] HybridClock initialized: now 1750751986406157 us; error 58 us; skew 500 ppm
I20250624 07:59:46.407019 23714 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:46.417886 23714 webserver.cc:469] Webserver started at http://127.17.219.254:43647/ using document root <none> and password file <none>
I20250624 07:59:46.418897 23714 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:46.419090 23714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:46.419625 23714 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:46.426294 23714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b6b71374f84e4a748d5346c35c6bb98b"
format_stamp: "Formatted at 2025-06-24 07:59:46 on dist-test-slave-qhgl"
I20250624 07:59:46.427460 23714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b6b71374f84e4a748d5346c35c6bb98b"
format_stamp: "Formatted at 2025-06-24 07:59:46 on dist-test-slave-qhgl"
I20250624 07:59:46.435530 23714 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250624 07:59:46.441944 23730 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:46.443197 23714 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:59:46.443527 23714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b6b71374f84e4a748d5346c35c6bb98b"
format_stamp: "Formatted at 2025-06-24 07:59:46 on dist-test-slave-qhgl"
I20250624 07:59:46.443871 23714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:46.489323 23714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:46.490978 23714 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:46.491442 23714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:46.572060 23714 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:45509
I20250624 07:59:46.572170 23781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:45509 every 8 connection(s)
I20250624 07:59:46.574960 23714 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 07:59:46.580044 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23714
I20250624 07:59:46.580492 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 07:59:46.580672 23782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:46.606925 23782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b: Bootstrap starting.
I20250624 07:59:46.612636 23782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:46.614317 23782 log.cc:826] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:46.619148 23782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b: No bootstrap required, opened a new log
I20250624 07:59:46.636749 23782 raft_consensus.cc:357] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b71374f84e4a748d5346c35c6bb98b" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 45509 } }
I20250624 07:59:46.637420 23782 raft_consensus.cc:383] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:46.637724 23782 raft_consensus.cc:738] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b71374f84e4a748d5346c35c6bb98b, State: Initialized, Role: FOLLOWER
I20250624 07:59:46.638590 23782 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b71374f84e4a748d5346c35c6bb98b" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 45509 } }
I20250624 07:59:46.639240 23782 raft_consensus.cc:397] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:59:46.639534 23782 raft_consensus.cc:491] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:59:46.639829 23782 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:46.643867 23782 raft_consensus.cc:513] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b71374f84e4a748d5346c35c6bb98b" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 45509 } }
I20250624 07:59:46.644565 23782 leader_election.cc:304] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6b71374f84e4a748d5346c35c6bb98b; no voters:
I20250624 07:59:46.646164 23782 leader_election.cc:290] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:59:46.647069 23787 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:59:46.649776 23787 raft_consensus.cc:695] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [term 1 LEADER]: Becoming Leader. State: Replica: b6b71374f84e4a748d5346c35c6bb98b, State: Running, Role: LEADER
I20250624 07:59:46.650609 23787 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b71374f84e4a748d5346c35c6bb98b" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 45509 } }
I20250624 07:59:46.651093 23782 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:59:46.661789 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6b71374f84e4a748d5346c35c6bb98b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b71374f84e4a748d5346c35c6bb98b" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 45509 } } }
I20250624 07:59:46.662905 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [sys.catalog]: This master's current role is: LEADER
I20250624 07:59:46.662111 23789 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6b71374f84e4a748d5346c35c6bb98b. Latest consensus state: current_term: 1 leader_uuid: "b6b71374f84e4a748d5346c35c6bb98b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b71374f84e4a748d5346c35c6bb98b" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 45509 } } }
I20250624 07:59:46.664371 23789 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b [sys.catalog]: This master's current role is: LEADER
I20250624 07:59:46.666240 23796 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:59:46.677501 23796 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:59:46.692123 23796 catalog_manager.cc:1349] Generated new cluster ID: 2991d8a22f444a039e7ce003ec646ee2
I20250624 07:59:46.692448 23796 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:59:46.726452 23796 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:59:46.728307 23796 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:59:46.741745 23796 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b6b71374f84e4a748d5346c35c6bb98b: Generated new TSK 0
I20250624 07:59:46.742569 23796 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:59:46.752250 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:45509
--builtin_ntp_servers=127.17.219.212:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 07:59:47.063562 23806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:47.064075 23806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:47.064579 23806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:47.096131 23806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:47.097010 23806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:59:47.132002 23806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:45509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:47.133291 23806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:47.134943 23806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:47.151485 23813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:47.154664 23806 server_base.cc:1048] running on GCE node
W20250624 07:59:47.154199 23815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:47.152621 23812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:48.355821 23806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:48.358513 23806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:48.360014 23806 hybrid_clock.cc:648] HybridClock initialized: now 1750751988359991 us; error 50 us; skew 500 ppm
I20250624 07:59:48.361076 23806 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:48.369223 23806 webserver.cc:469] Webserver started at http://127.17.219.193:40621/ using document root <none> and password file <none>
I20250624 07:59:48.370615 23806 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:48.370929 23806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:48.371547 23806 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:48.378341 23806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b1c0973be8c24958ae245edb81214038"
format_stamp: "Formatted at 2025-06-24 07:59:48 on dist-test-slave-qhgl"
I20250624 07:59:48.379843 23806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b1c0973be8c24958ae245edb81214038"
format_stamp: "Formatted at 2025-06-24 07:59:48 on dist-test-slave-qhgl"
I20250624 07:59:48.388870 23806 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20250624 07:59:48.395915 23822 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:48.397049 23806 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 07:59:48.397403 23806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b1c0973be8c24958ae245edb81214038"
format_stamp: "Formatted at 2025-06-24 07:59:48 on dist-test-slave-qhgl"
I20250624 07:59:48.397810 23806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:48.455176 23806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:48.456557 23806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:48.456951 23806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:48.459381 23806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:48.463363 23806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:48.463560 23806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:48.463766 23806 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:48.463899 23806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:48.598053 23806 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:33135
I20250624 07:59:48.598157 23934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:33135 every 8 connection(s)
I20250624 07:59:48.600589 23806 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 07:59:48.603133 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23806
I20250624 07:59:48.603752 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750751842019710-18287-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 07:59:48.620790 23935 heartbeater.cc:344] Connected to a master server at 127.17.219.254:45509
I20250624 07:59:48.621222 23935 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:48.622371 23935 heartbeater.cc:507] Master 127.17.219.254:45509 requested a full tablet report, sending...
I20250624 07:59:48.624781 23747 ts_manager.cc:194] Registered new tserver with Master: b1c0973be8c24958ae245edb81214038 (127.17.219.193:33135)
I20250624 07:59:48.626688 23747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:41731
I20250624 07:59:48.636862 18287 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250624 07:59:48.665717 23747 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43908:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 07:59:48.720727 23870 tablet_service.cc:1468] Processing CreateTablet for tablet 08661deadc6d4b09a1866761e7728626 (DEFAULT_TABLE table=TestTable [id=2d94dd348ca34c2d8b5caec140602c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:48.722092 23870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08661deadc6d4b09a1866761e7728626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:48.740617 23950 tablet_bootstrap.cc:492] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038: Bootstrap starting.
I20250624 07:59:48.745707 23950 tablet_bootstrap.cc:654] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:48.747412 23950 log.cc:826] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:48.751668 23950 tablet_bootstrap.cc:492] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038: No bootstrap required, opened a new log
I20250624 07:59:48.752039 23950 ts_tablet_manager.cc:1397] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20250624 07:59:48.768872 23950 raft_consensus.cc:357] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c0973be8c24958ae245edb81214038" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 33135 } }
I20250624 07:59:48.769398 23950 raft_consensus.cc:383] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:48.769654 23950 raft_consensus.cc:738] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c0973be8c24958ae245edb81214038, State: Initialized, Role: FOLLOWER
I20250624 07:59:48.770285 23950 consensus_queue.cc:260] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c0973be8c24958ae245edb81214038" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 33135 } }
I20250624 07:59:48.770802 23950 raft_consensus.cc:397] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:59:48.771031 23950 raft_consensus.cc:491] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:59:48.771281 23950 raft_consensus.cc:3058] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:48.775816 23950 raft_consensus.cc:513] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c0973be8c24958ae245edb81214038" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 33135 } }
I20250624 07:59:48.776480 23950 leader_election.cc:304] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1c0973be8c24958ae245edb81214038; no voters:
I20250624 07:59:48.778179 23950 leader_election.cc:290] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:59:48.778954 23952 raft_consensus.cc:2802] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:59:48.781176 23952 raft_consensus.cc:695] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [term 1 LEADER]: Becoming Leader. State: Replica: b1c0973be8c24958ae245edb81214038, State: Running, Role: LEADER
I20250624 07:59:48.781528 23935 heartbeater.cc:499] Master 127.17.219.254:45509 was elected leader, sending a full tablet report...
I20250624 07:59:48.782034 23952 consensus_queue.cc:237] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c0973be8c24958ae245edb81214038" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 33135 } }
I20250624 07:59:48.782492 23950 ts_tablet_manager.cc:1428] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038: Time spent starting tablet: real 0.030s user 0.027s sys 0.003s
I20250624 07:59:48.794152 23746 catalog_manager.cc:5582] T 08661deadc6d4b09a1866761e7728626 P b1c0973be8c24958ae245edb81214038 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1c0973be8c24958ae245edb81214038 (127.17.219.193). New cstate: current_term: 1 leader_uuid: "b1c0973be8c24958ae245edb81214038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c0973be8c24958ae245edb81214038" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 33135 } health_report { overall_health: HEALTHY } } }
I20250624 07:59:48.819574 18287 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250624 07:59:48.822331 18287 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b1c0973be8c24958ae245edb81214038 to finish bootstrapping
I20250624 07:59:51.528496 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23806
I20250624 07:59:51.555801 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23714
2025-06-24T07:59:51Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6842 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6843 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250624 07:59:51.618510 18287 test_util.cc:276] Using random seed: -1232282706
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-24T07:59:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:59:51Z Disabled control of system clock
I20250624 07:59:51.680940 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:41965
--webserver_interface=127.17.219.254
--webserver_port=0
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:41965 with env {}
W20250624 07:59:51.993304 23978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:51.993921 23978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:51.994367 23978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:52.025241 23978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:59:52.025553 23978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:52.025813 23978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:59:52.026046 23978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:59:52.060927 23978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:41965
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:52.062223 23978 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:52.063839 23978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:52.077386 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:52.078193 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:52.079733 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:52.080384 23978 server_base.cc:1048] running on GCE node
I20250624 07:59:53.262598 23978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:53.265702 23978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:53.267131 23978 hybrid_clock.cc:648] HybridClock initialized: now 1750751993267095 us; error 56 us; skew 500 ppm
I20250624 07:59:53.267931 23978 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:53.278468 23978 webserver.cc:469] Webserver started at http://127.17.219.254:45363/ using document root <none> and password file <none>
I20250624 07:59:53.279438 23978 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:53.279666 23978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:53.280210 23978 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:53.284610 23978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/instance:
uuid: "97cb79c336774a5b9f4a674b20a8ac6c"
format_stamp: "Formatted at 2025-06-24 07:59:53 on dist-test-slave-qhgl"
I20250624 07:59:53.285741 23978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal/instance:
uuid: "97cb79c336774a5b9f4a674b20a8ac6c"
format_stamp: "Formatted at 2025-06-24 07:59:53 on dist-test-slave-qhgl"
I20250624 07:59:53.292811 23978 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250624 07:59:53.298162 23995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:53.299149 23978 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:59:53.299454 23978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
uuid: "97cb79c336774a5b9f4a674b20a8ac6c"
format_stamp: "Formatted at 2025-06-24 07:59:53 on dist-test-slave-qhgl"
I20250624 07:59:53.299793 23978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:53.362532 23978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:53.364006 23978 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:53.364427 23978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:53.433391 23978 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:41965
I20250624 07:59:53.433465 24046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:41965 every 8 connection(s)
I20250624 07:59:53.436375 23978 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb
I20250624 07:59:53.441260 24047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:53.443753 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 23978
I20250624 07:59:53.444314 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal/instance
I20250624 07:59:53.462440 24047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c: Bootstrap starting.
I20250624 07:59:53.468497 24047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:53.470127 24047 log.cc:826] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:53.474462 24047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c: No bootstrap required, opened a new log
I20250624 07:59:53.492659 24047 raft_consensus.cc:357] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 07:59:53.493454 24047 raft_consensus.cc:383] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:53.493718 24047 raft_consensus.cc:738] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cb79c336774a5b9f4a674b20a8ac6c, State: Initialized, Role: FOLLOWER
I20250624 07:59:53.494354 24047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 07:59:53.494891 24047 raft_consensus.cc:397] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 07:59:53.495170 24047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 07:59:53.495434 24047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:59:53.499652 24047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 07:59:53.500411 24047 leader_election.cc:304] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97cb79c336774a5b9f4a674b20a8ac6c; no voters:
I20250624 07:59:53.501981 24047 leader_election.cc:290] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 07:59:53.502693 24052 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:59:53.504845 24052 raft_consensus.cc:695] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [term 1 LEADER]: Becoming Leader. State: Replica: 97cb79c336774a5b9f4a674b20a8ac6c, State: Running, Role: LEADER
I20250624 07:59:53.505558 24052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 07:59:53.506525 24047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:59:53.515290 24054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cb79c336774a5b9f4a674b20a8ac6c. Latest consensus state: current_term: 1 leader_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } } }
I20250624 07:59:53.515919 24054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [sys.catalog]: This master's current role is: LEADER
I20250624 07:59:53.517045 24053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cb79c336774a5b9f4a674b20a8ac6c" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } } }
I20250624 07:59:53.517489 24053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c [sys.catalog]: This master's current role is: LEADER
I20250624 07:59:53.519335 24059 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:59:53.531010 24059 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:59:53.549466 24059 catalog_manager.cc:1349] Generated new cluster ID: ddc5b09d80d646d2b8eb2334db6441c3
I20250624 07:59:53.549902 24059 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:59:53.571554 24059 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:59:53.573467 24059 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:59:53.593019 24059 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 97cb79c336774a5b9f4a674b20a8ac6c: Generated new TSK 0
I20250624 07:59:53.593868 24059 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:59:53.616855 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:0
--local_ip_for_outbound_sockets=127.17.219.193
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:41965
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250624 07:59:53.924113 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:53.924584 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:53.925072 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:53.956393 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:53.957242 24071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 07:59:53.993745 24071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=0
--tserver_master_addrs=127.17.219.254:41965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:53.995126 24071 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:53.996685 24071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:54.013489 24077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:54.013816 24078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:54.018100 24080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:55.534619 24079 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1520 milliseconds
I20250624 07:59:55.534785 24071 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:55.535913 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:55.538038 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:55.539362 24071 hybrid_clock.cc:648] HybridClock initialized: now 1750751995539324 us; error 55 us; skew 500 ppm
I20250624 07:59:55.540129 24071 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:55.546013 24071 webserver.cc:469] Webserver started at http://127.17.219.193:41741/ using document root <none> and password file <none>
I20250624 07:59:55.546941 24071 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:55.547132 24071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:55.547542 24071 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:55.551837 24071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/instance:
uuid: "34ed987cd3d3458f800bfe9b7d1fc94d"
format_stamp: "Formatted at 2025-06-24 07:59:55 on dist-test-slave-qhgl"
I20250624 07:59:55.552870 24071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal/instance:
uuid: "34ed987cd3d3458f800bfe9b7d1fc94d"
format_stamp: "Formatted at 2025-06-24 07:59:55 on dist-test-slave-qhgl"
I20250624 07:59:55.559736 24071 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250624 07:59:55.565243 24087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:55.566223 24071 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250624 07:59:55.566506 24071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
uuid: "34ed987cd3d3458f800bfe9b7d1fc94d"
format_stamp: "Formatted at 2025-06-24 07:59:55 on dist-test-slave-qhgl"
I20250624 07:59:55.566835 24071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:55.615360 24071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:55.616734 24071 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:55.617134 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:55.619517 24071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:55.623570 24071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:55.623760 24071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:55.623951 24071 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:55.624097 24071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:55.754907 24071 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:40815
I20250624 07:59:55.755014 24199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:40815 every 8 connection(s)
I20250624 07:59:55.757367 24071 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb
I20250624 07:59:55.764127 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24071
I20250624 07:59:55.764514 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal/instance
I20250624 07:59:55.770445 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:0
--local_ip_for_outbound_sockets=127.17.219.194
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:41965
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 07:59:55.778162 24200 heartbeater.cc:344] Connected to a master server at 127.17.219.254:41965
I20250624 07:59:55.778548 24200 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:55.779598 24200 heartbeater.cc:507] Master 127.17.219.254:41965 requested a full tablet report, sending...
I20250624 07:59:55.782230 24012 ts_manager.cc:194] Registered new tserver with Master: 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815)
I20250624 07:59:55.784157 24012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:45367
W20250624 07:59:56.071523 24204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:56.072026 24204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:56.072504 24204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:56.103681 24204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:56.104516 24204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 07:59:56.138532 24204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=0
--tserver_master_addrs=127.17.219.254:41965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:56.139873 24204 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:56.141425 24204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:59:56.157061 24211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:56.787930 24200 heartbeater.cc:499] Master 127.17.219.254:41965 was elected leader, sending a full tablet report...
W20250624 07:59:56.157459 24213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:56.157107 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:57.331594 24212 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1173 milliseconds
I20250624 07:59:57.331709 24204 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 07:59:57.332993 24204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:57.335186 24204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:57.336524 24204 hybrid_clock.cc:648] HybridClock initialized: now 1750751997336506 us; error 44 us; skew 500 ppm
I20250624 07:59:57.337303 24204 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:57.343468 24204 webserver.cc:469] Webserver started at http://127.17.219.194:39687/ using document root <none> and password file <none>
I20250624 07:59:57.344383 24204 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:57.344595 24204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:57.345052 24204 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:57.349506 24204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/instance:
uuid: "55d2201595184d50b969ed31c963640e"
format_stamp: "Formatted at 2025-06-24 07:59:57 on dist-test-slave-qhgl"
I20250624 07:59:57.350569 24204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal/instance:
uuid: "55d2201595184d50b969ed31c963640e"
format_stamp: "Formatted at 2025-06-24 07:59:57 on dist-test-slave-qhgl"
I20250624 07:59:57.357435 24204 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250624 07:59:57.362831 24220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:57.363817 24204 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:59:57.364120 24204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
uuid: "55d2201595184d50b969ed31c963640e"
format_stamp: "Formatted at 2025-06-24 07:59:57 on dist-test-slave-qhgl"
I20250624 07:59:57.364449 24204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:57.419143 24204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:57.420631 24204 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:57.421051 24204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:57.423552 24204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:57.427598 24204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:57.427809 24204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:57.428037 24204 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:57.428206 24204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:57.563491 24204 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:46855
I20250624 07:59:57.563581 24332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:46855 every 8 connection(s)
I20250624 07:59:57.565935 24204 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb
I20250624 07:59:57.573817 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24204
I20250624 07:59:57.574337 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal/instance
I20250624 07:59:57.581736 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:0
--local_ip_for_outbound_sockets=127.17.219.195
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:41965
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 07:59:57.588864 24333 heartbeater.cc:344] Connected to a master server at 127.17.219.254:41965
I20250624 07:59:57.589282 24333 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:57.590243 24333 heartbeater.cc:507] Master 127.17.219.254:41965 requested a full tablet report, sending...
I20250624 07:59:57.592427 24012 ts_manager.cc:194] Registered new tserver with Master: 55d2201595184d50b969ed31c963640e (127.17.219.194:46855)
I20250624 07:59:57.593618 24012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:47471
W20250624 07:59:57.879233 24337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:59:57.879725 24337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:59:57.880234 24337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:59:57.912768 24337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:59:57.913616 24337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 07:59:57.952224 24337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=0
--tserver_master_addrs=127.17.219.254:41965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 07:59:57.953552 24337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:59:57.955199 24337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250624 07:59:58.596732 24333 heartbeater.cc:499] Master 127.17.219.254:41965 was elected leader, sending a full tablet report...
I20250624 07:59:57.974543 24337 server_base.cc:1048] running on GCE node
W20250624 07:59:57.972297 24344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:57.972483 24346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:59:57.972286 24343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:59:59.131322 24337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:59:59.133543 24337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:59:59.134910 24337 hybrid_clock.cc:648] HybridClock initialized: now 1750751999134854 us; error 64 us; skew 500 ppm
I20250624 07:59:59.135691 24337 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:59:59.144826 24337 webserver.cc:469] Webserver started at http://127.17.219.195:44625/ using document root <none> and password file <none>
I20250624 07:59:59.145730 24337 fs_manager.cc:362] Metadata directory not provided
I20250624 07:59:59.145941 24337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:59:59.146384 24337 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:59:59.150820 24337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/instance:
uuid: "b9544347142c4f3b8e54ad201721bcf7"
format_stamp: "Formatted at 2025-06-24 07:59:59 on dist-test-slave-qhgl"
I20250624 07:59:59.151858 24337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal/instance:
uuid: "b9544347142c4f3b8e54ad201721bcf7"
format_stamp: "Formatted at 2025-06-24 07:59:59 on dist-test-slave-qhgl"
I20250624 07:59:59.158707 24337 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250624 07:59:59.164096 24353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:59.165063 24337 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250624 07:59:59.165359 24337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
uuid: "b9544347142c4f3b8e54ad201721bcf7"
format_stamp: "Formatted at 2025-06-24 07:59:59 on dist-test-slave-qhgl"
I20250624 07:59:59.165674 24337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:59:59.217274 24337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:59:59.218701 24337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:59:59.219133 24337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:59:59.221613 24337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:59:59.225693 24337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:59:59.225893 24337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:59.226135 24337 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:59:59.226313 24337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:59:59.358217 24337 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:41485
I20250624 07:59:59.358317 24465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:41485 every 8 connection(s)
I20250624 07:59:59.360721 24337 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb
I20250624 07:59:59.364192 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24337
I20250624 07:59:59.364617 18287 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal/instance
I20250624 07:59:59.380779 24466 heartbeater.cc:344] Connected to a master server at 127.17.219.254:41965
I20250624 07:59:59.381156 24466 heartbeater.cc:461] Registering TS with master...
I20250624 07:59:59.382167 24466 heartbeater.cc:507] Master 127.17.219.254:41965 requested a full tablet report, sending...
I20250624 07:59:59.384325 24012 ts_manager.cc:194] Registered new tserver with Master: b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485)
I20250624 07:59:59.385627 24012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:50049
I20250624 07:59:59.398741 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:59:59.427857 18287 test_util.cc:276] Using random seed: -1224473350
I20250624 07:59:59.467285 24012 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43336:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 07:59:59.469513 24012 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:59:59.521546 24268 tablet_service.cc:1468] Processing CreateTablet for tablet 4eb45be406ce41bfba3af333c4cac366 (DEFAULT_TABLE table=pre_rebuild [id=fe6be6834f6342929248c08638b1474e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:59.523481 24268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb45be406ce41bfba3af333c4cac366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:59.523664 24401 tablet_service.cc:1468] Processing CreateTablet for tablet 4eb45be406ce41bfba3af333c4cac366 (DEFAULT_TABLE table=pre_rebuild [id=fe6be6834f6342929248c08638b1474e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:59.524727 24135 tablet_service.cc:1468] Processing CreateTablet for tablet 4eb45be406ce41bfba3af333c4cac366 (DEFAULT_TABLE table=pre_rebuild [id=fe6be6834f6342929248c08638b1474e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:59:59.525552 24401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb45be406ce41bfba3af333c4cac366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:59.526489 24135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb45be406ce41bfba3af333c4cac366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:59:59.544572 24490 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Bootstrap starting.
I20250624 07:59:59.550067 24490 tablet_bootstrap.cc:654] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:59.552117 24490 log.cc:826] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:59.553907 24491 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Bootstrap starting.
I20250624 07:59:59.555760 24492 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Bootstrap starting.
I20250624 07:59:59.559885 24490 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: No bootstrap required, opened a new log
I20250624 07:59:59.560456 24490 ts_tablet_manager.cc:1397] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20250624 07:59:59.561187 24491 tablet_bootstrap.cc:654] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:59.563076 24492 tablet_bootstrap.cc:654] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Neither blocks nor log segments found. Creating new log.
I20250624 07:59:59.563755 24491 log.cc:826] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:59.565378 24492 log.cc:826] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Log is configured to *not* fsync() on all Append() calls
I20250624 07:59:59.576612 24492 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: No bootstrap required, opened a new log
I20250624 07:59:59.577265 24492 ts_tablet_manager.cc:1397] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.018s
I20250624 07:59:59.586769 24491 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: No bootstrap required, opened a new log
I20250624 07:59:59.587359 24491 ts_tablet_manager.cc:1397] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20250624 07:59:59.596143 24490 raft_consensus.cc:357] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.597203 24490 raft_consensus.cc:383] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:59.597538 24490 raft_consensus.cc:738] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d2201595184d50b969ed31c963640e, State: Initialized, Role: FOLLOWER
I20250624 07:59:59.598702 24490 consensus_queue.cc:260] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.610549 24492 raft_consensus.cc:357] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.611580 24490 ts_tablet_manager.cc:1428] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Time spent starting tablet: real 0.051s user 0.034s sys 0.004s
I20250624 07:59:59.611637 24492 raft_consensus.cc:383] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:59.611995 24492 raft_consensus.cc:738] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9544347142c4f3b8e54ad201721bcf7, State: Initialized, Role: FOLLOWER
I20250624 07:59:59.613093 24492 consensus_queue.cc:260] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.614216 24491 raft_consensus.cc:357] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.615151 24491 raft_consensus.cc:383] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:59:59.615460 24491 raft_consensus.cc:738] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ed987cd3d3458f800bfe9b7d1fc94d, State: Initialized, Role: FOLLOWER
I20250624 07:59:59.616416 24491 consensus_queue.cc:260] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.622120 24466 heartbeater.cc:499] Master 127.17.219.254:41965 was elected leader, sending a full tablet report...
I20250624 07:59:59.624974 24491 ts_tablet_manager.cc:1428] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Time spent starting tablet: real 0.037s user 0.026s sys 0.009s
I20250624 07:59:59.626881 24492 ts_tablet_manager.cc:1428] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Time spent starting tablet: real 0.049s user 0.033s sys 0.004s
W20250624 07:59:59.766525 24201 tablet.cc:2378] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:59:59.823534 24334 tablet.cc:2378] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 07:59:59.865943 24467 tablet.cc:2378] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:59:59.989519 24496 raft_consensus.cc:491] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:59:59.990006 24496 raft_consensus.cc:513] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 07:59:59.992552 24496 leader_election.cc:290] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485), 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815)
I20250624 08:00:00.003851 24421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "55d2201595184d50b969ed31c963640e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9544347142c4f3b8e54ad201721bcf7" is_pre_election: true
I20250624 08:00:00.003886 24155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "55d2201595184d50b969ed31c963640e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" is_pre_election: true
I20250624 08:00:00.004525 24421 raft_consensus.cc:2466] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d2201595184d50b969ed31c963640e in term 0.
I20250624 08:00:00.004568 24155 raft_consensus.cc:2466] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d2201595184d50b969ed31c963640e in term 0.
I20250624 08:00:00.005730 24223 leader_election.cc:304] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d2201595184d50b969ed31c963640e, b9544347142c4f3b8e54ad201721bcf7; no voters:
I20250624 08:00:00.006532 24496 raft_consensus.cc:2802] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:00:00.006850 24496 raft_consensus.cc:491] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:00:00.007117 24496 raft_consensus.cc:3058] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:00.011684 24496 raft_consensus.cc:513] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:00.013028 24496 leader_election.cc:290] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [CANDIDATE]: Term 1 election: Requested vote from peers b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485), 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815)
I20250624 08:00:00.013912 24421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "55d2201595184d50b969ed31c963640e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9544347142c4f3b8e54ad201721bcf7"
I20250624 08:00:00.013949 24155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "55d2201595184d50b969ed31c963640e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d"
I20250624 08:00:00.014357 24155 raft_consensus.cc:3058] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:00.014357 24421 raft_consensus.cc:3058] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:00.018751 24155 raft_consensus.cc:2466] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d2201595184d50b969ed31c963640e in term 1.
I20250624 08:00:00.019078 24421 raft_consensus.cc:2466] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d2201595184d50b969ed31c963640e in term 1.
I20250624 08:00:00.020085 24224 leader_election.cc:304] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ed987cd3d3458f800bfe9b7d1fc94d, 55d2201595184d50b969ed31c963640e; no voters:
I20250624 08:00:00.020676 24496 raft_consensus.cc:2802] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:00:00.021906 24496 raft_consensus.cc:695] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 LEADER]: Becoming Leader. State: Replica: 55d2201595184d50b969ed31c963640e, State: Running, Role: LEADER
I20250624 08:00:00.022624 24496 consensus_queue.cc:237] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:00.033535 24012 catalog_manager.cc:5582] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e reported cstate change: term changed from 0 to 1, leader changed from <none> to 55d2201595184d50b969ed31c963640e (127.17.219.194). New cstate: current_term: 1 leader_uuid: "55d2201595184d50b969ed31c963640e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } health_report { overall_health: UNKNOWN } } }
I20250624 08:00:00.196556 24155 raft_consensus.cc:1273] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Refusing update from remote peer 55d2201595184d50b969ed31c963640e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:00:00.196712 24421 raft_consensus.cc:1273] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Refusing update from remote peer 55d2201595184d50b969ed31c963640e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:00:00.198184 24496 consensus_queue.cc:1035] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:00:00.198779 24501 consensus_queue.cc:1035] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:00:00.229987 24517 mvcc.cc:204] Tried to move back new op lower bound from 7171080192790167552 to 7171080192099319808. Current Snapshot: MvccSnapshot[applied={T|T < 7171080192790167552}]
I20250624 08:00:00.231885 24513 mvcc.cc:204] Tried to move back new op lower bound from 7171080192810205184 to 7171080192099319808. Current Snapshot: MvccSnapshot[applied={T|T < 7171080192790167552}]
I20250624 08:00:00.260304 24518 mvcc.cc:204] Tried to move back new op lower bound from 7171080192790167552 to 7171080192099319808. Current Snapshot: MvccSnapshot[applied={T|T < 7171080192790167552}]
I20250624 08:00:05.320405 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 23978
W20250624 08:00:05.359746 24200 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:41965 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:41965: connect: Connection refused (error 111)
W20250624 08:00:05.677718 24554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:00:05.678336 24554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:00:05.713291 24554 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 08:00:06.309541 24466 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:41965 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:41965: connect: Connection refused (error 111)
W20250624 08:00:06.313138 24333 heartbeater.cc:646] Failed to heartbeat to 127.17.219.254:41965 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.219.254:41965: connect: Connection refused (error 111)
W20250624 08:00:07.162137 24559 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 24554
W20250624 08:00:07.162739 24559 kernel_stack_watchdog.cc:198] Thread 24554 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250624 08:00:07.174337 24554 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.415s user 0.002s sys 0.000s
W20250624 08:00:07.287681 24554 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.528s user 0.005s sys 0.012s
I20250624 08:00:07.388700 24554 minidump.cc:252] Setting minidump size limit to 20M
I20250624 08:00:07.392326 24554 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:00:07.393707 24554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:00:07.408120 24592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:00:07.408521 24591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:00:07.410054 24594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:07.410385 24554 server_base.cc:1048] running on GCE node
I20250624 08:00:07.411533 24554 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250624 08:00:07.411960 24554 hybrid_clock.cc:648] HybridClock initialized: now 1750752007411938 us; error 282516 us; skew 500 ppm
I20250624 08:00:07.412672 24554 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 08:00:07.417357 24554 webserver.cc:469] Webserver started at http://0.0.0.0:46857/ using document root <none> and password file <none>
I20250624 08:00:07.418165 24554 fs_manager.cc:362] Metadata directory not provided
I20250624 08:00:07.418394 24554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:00:07.418854 24554 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:00:07.422993 24554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/instance:
uuid: "59cafb09591c4e1680cd91853eb1b3b3"
format_stamp: "Formatted at 2025-06-24 08:00:07 on dist-test-slave-qhgl"
I20250624 08:00:07.424055 24554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal/instance:
uuid: "59cafb09591c4e1680cd91853eb1b3b3"
format_stamp: "Formatted at 2025-06-24 08:00:07 on dist-test-slave-qhgl"
I20250624 08:00:07.431042 24554 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250624 08:00:07.435707 24599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:00:07.436627 24554 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:00:07.436926 24554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
uuid: "59cafb09591c4e1680cd91853eb1b3b3"
format_stamp: "Formatted at 2025-06-24 08:00:07 on dist-test-slave-qhgl"
I20250624 08:00:07.437224 24554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:00:07.587975 24554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:00:07.589322 24554 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:00:07.589761 24554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:00:07.594231 24554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:00:07.608310 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Bootstrap starting.
I20250624 08:00:07.613201 24554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Neither blocks nor log segments found. Creating new log.
I20250624 08:00:07.614799 24554 log.cc:826] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Log is configured to *not* fsync() on all Append() calls
I20250624 08:00:07.618620 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: No bootstrap required, opened a new log
I20250624 08:00:07.635196 24554 raft_consensus.cc:357] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER }
I20250624 08:00:07.635663 24554 raft_consensus.cc:383] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:00:07.635903 24554 raft_consensus.cc:738] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cafb09591c4e1680cd91853eb1b3b3, State: Initialized, Role: FOLLOWER
I20250624 08:00:07.636581 24554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER }
I20250624 08:00:07.637024 24554 raft_consensus.cc:397] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:00:07.637267 24554 raft_consensus.cc:491] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:00:07.637555 24554 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:07.641547 24554 raft_consensus.cc:513] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER }
I20250624 08:00:07.642149 24554 leader_election.cc:304] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59cafb09591c4e1680cd91853eb1b3b3; no voters:
I20250624 08:00:07.643857 24554 leader_election.cc:290] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:00:07.644090 24606 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:00:07.646168 24606 raft_consensus.cc:695] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 59cafb09591c4e1680cd91853eb1b3b3, State: Running, Role: LEADER
I20250624 08:00:07.647076 24606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER }
I20250624 08:00:07.654539 24607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59cafb09591c4e1680cd91853eb1b3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER } }
I20250624 08:00:07.655153 24607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: This master's current role is: LEADER
I20250624 08:00:07.655117 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59cafb09591c4e1680cd91853eb1b3b3. Latest consensus state: current_term: 1 leader_uuid: "59cafb09591c4e1680cd91853eb1b3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER } }
I20250624 08:00:07.655714 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: This master's current role is: LEADER
I20250624 08:00:07.665639 24554 tablet_replica.cc:331] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: stopping tablet replica
I20250624 08:00:07.666095 24554 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 LEADER]: Raft consensus shutting down.
I20250624 08:00:07.666435 24554 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 08:00:07.668323 24554 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250624 08:00:07.668720 24554 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250624 08:00:07.722815 24554 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250624 08:00:08.751684 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24071
W20250624 08:00:08.791308 24224 connection.cc:537] client connection to 127.17.219.193:40815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 08:00:08.791639 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24204
I20250624 08:00:08.833047 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24337
I20250624 08:00:08.867655 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:41965
--webserver_interface=127.17.219.254
--webserver_port=45363
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.219.254:41965 with env {}
W20250624 08:00:09.169154 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:00:09.169768 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:00:09.170233 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:00:09.201040 24616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:00:09.201364 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:00:09.201632 24616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:00:09.201874 24616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:00:09.237524 24616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.219.254:41965
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.219.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.219.254
--webserver_port=45363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 08:00:09.238876 24616 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:00:09.240502 24616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:00:09.255477 24623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:00:09.255784 24622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:00:09.255991 24625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:09.257962 24616 server_base.cc:1048] running on GCE node
I20250624 08:00:10.435731 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:00:10.438249 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:00:10.439615 24616 hybrid_clock.cc:648] HybridClock initialized: now 1750752010439589 us; error 42 us; skew 500 ppm
I20250624 08:00:10.440411 24616 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 08:00:10.446975 24616 webserver.cc:469] Webserver started at http://127.17.219.254:45363/ using document root <none> and password file <none>
I20250624 08:00:10.447841 24616 fs_manager.cc:362] Metadata directory not provided
I20250624 08:00:10.448037 24616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:00:10.455446 24616 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.002s
I20250624 08:00:10.459676 24632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:00:10.460625 24616 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250624 08:00:10.460896 24616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
uuid: "59cafb09591c4e1680cd91853eb1b3b3"
format_stamp: "Formatted at 2025-06-24 08:00:07 on dist-test-slave-qhgl"
I20250624 08:00:10.462760 24616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:00:10.512619 24616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:00:10.514037 24616 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:00:10.514433 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:00:10.583534 24616 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.254:41965
I20250624 08:00:10.583601 24683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.254:41965 every 8 connection(s)
I20250624 08:00:10.586350 24616 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb
I20250624 08:00:10.591125 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24616
I20250624 08:00:10.593173 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.193:40815
--local_ip_for_outbound_sockets=127.17.219.193
--tserver_master_addrs=127.17.219.254:41965
--webserver_port=41741
--webserver_interface=127.17.219.193
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:00:10.600093 24684 sys_catalog.cc:263] Verifying existing consensus state
I20250624 08:00:10.613755 24684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Bootstrap starting.
I20250624 08:00:10.623339 24684 log.cc:826] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Log is configured to *not* fsync() on all Append() calls
I20250624 08:00:10.634640 24684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:00:10.635499 24684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Bootstrap complete.
I20250624 08:00:10.655913 24684 raft_consensus.cc:357] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 08:00:10.656567 24684 raft_consensus.cc:738] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cafb09591c4e1680cd91853eb1b3b3, State: Initialized, Role: FOLLOWER
I20250624 08:00:10.657405 24684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 08:00:10.657918 24684 raft_consensus.cc:397] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:00:10.658174 24684 raft_consensus.cc:491] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:00:10.658459 24684 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:00:10.663218 24684 raft_consensus.cc:513] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 08:00:10.663892 24684 leader_election.cc:304] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59cafb09591c4e1680cd91853eb1b3b3; no voters:
I20250624 08:00:10.666036 24684 leader_election.cc:290] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 08:00:10.666360 24688 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:00:10.669690 24688 raft_consensus.cc:695] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [term 2 LEADER]: Becoming Leader. State: Replica: 59cafb09591c4e1680cd91853eb1b3b3, State: Running, Role: LEADER
I20250624 08:00:10.670526 24688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } }
I20250624 08:00:10.671285 24684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:00:10.680377 24689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "59cafb09591c4e1680cd91853eb1b3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } } }
I20250624 08:00:10.682301 24689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: This master's current role is: LEADER
I20250624 08:00:10.681633 24690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59cafb09591c4e1680cd91853eb1b3b3. Latest consensus state: current_term: 2 leader_uuid: "59cafb09591c4e1680cd91853eb1b3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cafb09591c4e1680cd91853eb1b3b3" member_type: VOTER last_known_addr { host: "127.17.219.254" port: 41965 } } }
I20250624 08:00:10.684329 24690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3 [sys.catalog]: This master's current role is: LEADER
I20250624 08:00:10.689630 24695 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:00:10.704120 24695 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=f1a812e91f294dbc9580bb060f652d18]
I20250624 08:00:10.712899 24695 tablet_loader.cc:96] loaded metadata for tablet 4eb45be406ce41bfba3af333c4cac366 (table pre_rebuild [id=f1a812e91f294dbc9580bb060f652d18])
I20250624 08:00:10.714798 24695 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:00:10.749854 24695 catalog_manager.cc:1349] Generated new cluster ID: 2fbb4fb17c3a4f159e347a63b07e166c
I20250624 08:00:10.750806 24695 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:00:10.766279 24695 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:00:10.767673 24695 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:00:10.781347 24695 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Generated new TSK 0
I20250624 08:00:10.782140 24695 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 08:00:10.957954 24686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:00:10.958575 24686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:00:10.959102 24686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:00:10.991029 24686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:00:10.991880 24686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.193
I20250624 08:00:11.026361 24686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.193:40815
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.219.193
--webserver_port=41741
--tserver_master_addrs=127.17.219.254:41965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.193
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 08:00:11.027679 24686 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:00:11.029253 24686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:00:11.046561 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:00:11.049551 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:00:11.048849 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:11.049921 24686 server_base.cc:1048] running on GCE node
I20250624 08:00:12.223141 24686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:00:12.225986 24686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:00:12.227538 24686 hybrid_clock.cc:648] HybridClock initialized: now 1750752012227498 us; error 44 us; skew 500 ppm
I20250624 08:00:12.228632 24686 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 08:00:12.236228 24686 webserver.cc:469] Webserver started at http://127.17.219.193:41741/ using document root <none> and password file <none>
I20250624 08:00:12.237447 24686 fs_manager.cc:362] Metadata directory not provided
I20250624 08:00:12.237772 24686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:00:12.248591 24686 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20250624 08:00:12.254438 24722 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:00:12.255702 24686 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250624 08:00:12.256083 24686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
uuid: "34ed987cd3d3458f800bfe9b7d1fc94d"
format_stamp: "Formatted at 2025-06-24 07:59:55 on dist-test-slave-qhgl"
I20250624 08:00:12.258765 24686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:00:12.329741 24686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:00:12.331708 24686 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:00:12.332249 24686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:00:12.335451 24686 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:00:12.342682 24729 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 08:00:12.350140 24686 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:00:12.350455 24686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.001s
I20250624 08:00:12.350814 24686 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:00:12.357941 24686 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:00:12.358209 24686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.004s sys 0.000s
I20250624 08:00:12.358608 24729 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Bootstrap starting.
I20250624 08:00:12.552249 24686 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.193:40815
I20250624 08:00:12.552573 24836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.193:40815 every 8 connection(s)
I20250624 08:00:12.555989 24686 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb
I20250624 08:00:12.563611 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24686
I20250624 08:00:12.565778 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.194:46855
--local_ip_for_outbound_sockets=127.17.219.194
--tserver_master_addrs=127.17.219.254:41965
--webserver_port=39687
--webserver_interface=127.17.219.194
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:00:12.623490 24837 heartbeater.cc:344] Connected to a master server at 127.17.219.254:41965
I20250624 08:00:12.623988 24837 heartbeater.cc:461] Registering TS with master...
I20250624 08:00:12.625156 24837 heartbeater.cc:507] Master 127.17.219.254:41965 requested a full tablet report, sending...
I20250624 08:00:12.630035 24649 ts_manager.cc:194] Registered new tserver with Master: 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815)
I20250624 08:00:12.637749 24649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.193:60765
I20250624 08:00:12.675452 24729 log.cc:826] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Log is configured to *not* fsync() on all Append() calls
W20250624 08:00:12.930606 24841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:00:12.931150 24841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:00:12.931630 24841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:00:12.962661 24841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:00:12.963557 24841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.194
I20250624 08:00:13.001209 24841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.194:46855
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.219.194
--webserver_port=39687
--tserver_master_addrs=127.17.219.254:41965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.194
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 08:00:13.002521 24841 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:00:13.004181 24841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:00:13.021299 24849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:13.642540 24837 heartbeater.cc:499] Master 127.17.219.254:41965 was elected leader, sending a full tablet report...
W20250624 08:00:13.024248 24851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:13.026247 24841 server_base.cc:1048] running on GCE node
W20250624 08:00:13.025178 24848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:14.218645 24841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:00:14.221577 24841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:00:14.223028 24841 hybrid_clock.cc:648] HybridClock initialized: now 1750752014222964 us; error 71 us; skew 500 ppm
I20250624 08:00:14.224124 24841 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 08:00:14.231624 24841 webserver.cc:469] Webserver started at http://127.17.219.194:39687/ using document root <none> and password file <none>
I20250624 08:00:14.232836 24841 fs_manager.cc:362] Metadata directory not provided
I20250624 08:00:14.233127 24841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:00:14.243558 24841 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250624 08:00:14.249426 24858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:00:14.250598 24841 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250624 08:00:14.251014 24841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
uuid: "55d2201595184d50b969ed31c963640e"
format_stamp: "Formatted at 2025-06-24 07:59:57 on dist-test-slave-qhgl"
I20250624 08:00:14.253710 24841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:00:14.311079 24841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:00:14.312548 24841 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:00:14.312984 24841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:00:14.315627 24841 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:00:14.322515 24865 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 08:00:14.329699 24841 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:00:14.329918 24841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.001s
I20250624 08:00:14.330204 24841 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:00:14.335003 24841 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:00:14.335273 24841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250624 08:00:14.335614 24865 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Bootstrap starting.
W20250624 08:00:14.543663 24680 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0280 after lost signal to thread 24617
W20250624 08:00:14.544472 24680 debug-util.cc:398] Leaking SignalData structure 0x7b080006f180 after lost signal to thread 24683
I20250624 08:00:14.616448 24971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.194:46855 every 8 connection(s)
I20250624 08:00:14.617089 24841 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.194:46855
I20250624 08:00:14.621209 24841 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb
I20250624 08:00:14.629705 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24841
I20250624 08:00:14.632014 18287 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
/tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.219.195:41485
--local_ip_for_outbound_sockets=127.17.219.195
--tserver_master_addrs=127.17.219.254:41965
--webserver_port=44625
--webserver_interface=127.17.219.195
--builtin_ntp_servers=127.17.219.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:00:14.662822 24865 log.cc:826] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Log is configured to *not* fsync() on all Append() calls
I20250624 08:00:14.696844 24972 heartbeater.cc:344] Connected to a master server at 127.17.219.254:41965
I20250624 08:00:14.697366 24972 heartbeater.cc:461] Registering TS with master...
I20250624 08:00:14.698666 24972 heartbeater.cc:507] Master 127.17.219.254:41965 requested a full tablet report, sending...
I20250624 08:00:14.702939 24649 ts_manager.cc:194] Registered new tserver with Master: 55d2201595184d50b969ed31c963640e (127.17.219.194:46855)
I20250624 08:00:14.706146 24649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.194:48569
I20250624 08:00:14.884502 24729 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Bootstrap replayed 1/1 log segments. Stats: ops{read=206 overwritten=0 applied=206 ignored=0} inserts{seen=10250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:00:14.885787 24729 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Bootstrap complete.
I20250624 08:00:14.887610 24729 ts_tablet_manager.cc:1397] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Time spent bootstrapping tablet: real 2.529s user 2.439s sys 0.083s
I20250624 08:00:14.904362 24729 raft_consensus.cc:357] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:14.907485 24729 raft_consensus.cc:738] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ed987cd3d3458f800bfe9b7d1fc94d, State: Initialized, Role: FOLLOWER
I20250624 08:00:14.908528 24729 consensus_queue.cc:260] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:14.917732 24729 ts_tablet_manager.cc:1428] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Time spent starting tablet: real 0.030s user 0.023s sys 0.004s
W20250624 08:00:15.162989 24976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:00:15.163625 24976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:00:15.164369 24976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:00:15.219516 24976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:00:15.220918 24976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.219.195
I20250624 08:00:15.281617 24976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.219.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.219.195:41485
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.219.195
--webserver_port=44625
--tserver_master_addrs=127.17.219.254:41965
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.219.195
--log_dir=/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6745
TSAN enabled
I20250624 08:00:15.283296 24976 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:00:15.285430 24976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:00:15.305836 24984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:15.710493 24972 heartbeater.cc:499] Master 127.17.219.254:41965 was elected leader, sending a full tablet report...
I20250624 08:00:16.077931 24988 raft_consensus.cc:491] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:00:16.078740 24988 raft_consensus.cc:513] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:16.116500 24988 leader_election.cc:290] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55d2201595184d50b969ed31c963640e (127.17.219.194:46855), b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485)
W20250624 08:00:16.130245 24725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.219.195:41485: connect: Connection refused (error 111)
I20250624 08:00:16.141279 24927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "55d2201595184d50b969ed31c963640e" is_pre_election: true
W20250624 08:00:16.157635 24725 leader_election.cc:343] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55d2201595184d50b969ed31c963640e (127.17.219.194:46855): Illegal state: must be running to vote when last-logged opid is not known
W20250624 08:00:16.160816 24725 leader_election.cc:336] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485): Network error: Client connection negotiation failed: client connection to 127.17.219.195:41485: connect: Connection refused (error 111)
I20250624 08:00:16.162053 24725 leader_election.cc:304] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34ed987cd3d3458f800bfe9b7d1fc94d; no voters: 55d2201595184d50b969ed31c963640e, b9544347142c4f3b8e54ad201721bcf7
I20250624 08:00:16.163846 24988 raft_consensus.cc:2747] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250624 08:00:16.701381 24983 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 24976
W20250624 08:00:16.799322 24976 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 1.495s user 0.487s sys 0.823s
W20250624 08:00:16.800699 24976 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 1.496s user 0.488s sys 0.824s
W20250624 08:00:15.306979 24985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:16.812650 24976 server_base.cc:1048] running on GCE node
W20250624 08:00:16.814409 24994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:00:16.815939 24976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:00:16.818521 24976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:00:16.820103 24976 hybrid_clock.cc:648] HybridClock initialized: now 1750752016820053 us; error 37 us; skew 500 ppm
I20250624 08:00:16.821336 24976 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 08:00:16.829372 24976 webserver.cc:469] Webserver started at http://127.17.219.195:44625/ using document root <none> and password file <none>
I20250624 08:00:16.830665 24976 fs_manager.cc:362] Metadata directory not provided
I20250624 08:00:16.830996 24976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:00:16.844563 24976 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20250624 08:00:16.851114 24999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:00:16.852582 24976 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250624 08:00:16.853006 24976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
uuid: "b9544347142c4f3b8e54ad201721bcf7"
format_stamp: "Formatted at 2025-06-24 07:59:59 on dist-test-slave-qhgl"
I20250624 08:00:16.855954 24976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:00:16.920962 24976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:00:16.923070 24976 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:00:16.923684 24976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:00:16.927346 24976 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:00:16.939947 25006 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 08:00:16.950992 24976 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:00:16.951335 24976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.003s sys 0.000s
I20250624 08:00:16.951689 24976 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:00:16.959882 24976 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:00:16.960245 24976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.009s sys 0.000s
I20250624 08:00:16.960557 25006 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Bootstrap starting.
I20250624 08:00:17.287736 25006 log.cc:826] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Log is configured to *not* fsync() on all Append() calls
I20250624 08:00:17.381780 24976 rpc_server.cc:307] RPC server started. Bound to: 127.17.219.195:41485
I20250624 08:00:17.382043 25113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.219.195:41485 every 8 connection(s)
I20250624 08:00:17.385335 24976 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb
I20250624 08:00:17.395787 18287 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu as pid 24976
I20250624 08:00:17.463230 25114 heartbeater.cc:344] Connected to a master server at 127.17.219.254:41965
I20250624 08:00:17.463761 25114 heartbeater.cc:461] Registering TS with master...
I20250624 08:00:17.465029 25114 heartbeater.cc:507] Master 127.17.219.254:41965 requested a full tablet report, sending...
I20250624 08:00:17.468860 24649 ts_manager.cc:194] Registered new tserver with Master: b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485)
I20250624 08:00:17.475113 24649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.219.195:44849
I20250624 08:00:17.481140 18287 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:00:17.899291 24865 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Bootstrap replayed 1/1 log segments. Stats: ops{read=206 overwritten=0 applied=206 ignored=0} inserts{seen=10250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:00:17.900104 24865 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Bootstrap complete.
I20250624 08:00:17.901419 24865 ts_tablet_manager.cc:1397] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Time spent bootstrapping tablet: real 3.566s user 3.405s sys 0.076s
I20250624 08:00:17.906780 24865 raft_consensus.cc:357] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:17.908730 24865 raft_consensus.cc:738] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d2201595184d50b969ed31c963640e, State: Initialized, Role: FOLLOWER
I20250624 08:00:17.909471 24865 consensus_queue.cc:260] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:17.912500 24865 ts_tablet_manager.cc:1428] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20250624 08:00:18.134348 25126 raft_consensus.cc:491] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:00:18.134843 25126 raft_consensus.cc:513] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:18.136667 25126 leader_election.cc:290] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55d2201595184d50b969ed31c963640e (127.17.219.194:46855), b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485)
I20250624 08:00:18.137480 24927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "55d2201595184d50b969ed31c963640e" is_pre_election: true
I20250624 08:00:18.138348 24927 raft_consensus.cc:2466] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ed987cd3d3458f800bfe9b7d1fc94d in term 1.
I20250624 08:00:18.140609 24725 leader_election.cc:304] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ed987cd3d3458f800bfe9b7d1fc94d, 55d2201595184d50b969ed31c963640e; no voters:
I20250624 08:00:18.141572 25126 raft_consensus.cc:2802] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 08:00:18.141927 25126 raft_consensus.cc:491] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:00:18.142271 25126 raft_consensus.cc:3058] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:00:18.149935 25126 raft_consensus.cc:513] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:18.151719 25126 leader_election.cc:290] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 election: Requested vote from peers 55d2201595184d50b969ed31c963640e (127.17.219.194:46855), b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485)
I20250624 08:00:18.152691 24927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "55d2201595184d50b969ed31c963640e"
I20250624 08:00:18.153167 24927 raft_consensus.cc:3058] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:00:18.159493 24927 raft_consensus.cc:2466] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ed987cd3d3458f800bfe9b7d1fc94d in term 2.
I20250624 08:00:18.160369 24725 leader_election.cc:304] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ed987cd3d3458f800bfe9b7d1fc94d, 55d2201595184d50b969ed31c963640e; no voters:
I20250624 08:00:18.160984 25126 raft_consensus.cc:2802] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:00:18.162757 25126 raft_consensus.cc:695] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 LEADER]: Becoming Leader. State: Replica: 34ed987cd3d3458f800bfe9b7d1fc94d, State: Running, Role: LEADER
I20250624 08:00:18.154341 25068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "b9544347142c4f3b8e54ad201721bcf7"
I20250624 08:00:18.152717 25069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eb45be406ce41bfba3af333c4cac366" candidate_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "b9544347142c4f3b8e54ad201721bcf7" is_pre_election: true
I20250624 08:00:18.163736 25126 consensus_queue.cc:237] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
W20250624 08:00:18.165603 24725 leader_election.cc:343] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485): Illegal state: must be running to vote when last-logged opid is not known
W20250624 08:00:18.166448 24725 leader_election.cc:343] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485): Illegal state: must be running to vote when last-logged opid is not known
I20250624 08:00:18.174678 24649 catalog_manager.cc:5582] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d reported cstate change: term changed from 0 to 2, leader changed from <none> to 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193), VOTER 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193) added, VOTER 55d2201595184d50b969ed31c963640e (127.17.219.194) added, VOTER b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195) added. New cstate: current_term: 2 leader_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } health_report { overall_health: HEALTHY } } }
W20250624 08:00:18.260116 24680 debug-util.cc:398] Leaking SignalData structure 0x7b080009e6c0 after lost signal to thread 24617
I20250624 08:00:18.479820 25114 heartbeater.cc:499] Master 127.17.219.254:41965 was elected leader, sending a full tablet report...
I20250624 08:00:18.728693 24927 raft_consensus.cc:1273] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 2 FOLLOWER]: Refusing update from remote peer 34ed987cd3d3458f800bfe9b7d1fc94d: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250624 08:00:18.730424 25139 consensus_queue.cc:1035] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
W20250624 08:00:18.800426 24725 consensus_peers.cc:487] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d -> Peer b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485): Couldn't send request to peer b9544347142c4f3b8e54ad201721bcf7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250624 08:00:18.833408 24791 consensus_queue.cc:237] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 206, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
W20250624 08:00:18.835374 18287 scanner-internal.cc:458] Time spent opening tablet: real 1.316s user 0.005s sys 0.001s
I20250624 08:00:18.839445 24927 raft_consensus.cc:1273] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 2 FOLLOWER]: Refusing update from remote peer 34ed987cd3d3458f800bfe9b7d1fc94d: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20250624 08:00:18.841099 25142 consensus_queue.cc:1035] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20250624 08:00:18.847513 25139 raft_consensus.cc:2953] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index -1 to 208, VOTER b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } } }
I20250624 08:00:18.857167 24927 raft_consensus.cc:2953] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index -1 to 208, VOTER b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } } }
I20250624 08:00:18.868920 24636 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4eb45be406ce41bfba3af333c4cac366 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 08:00:18.873853 24649 catalog_manager.cc:5582] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d reported cstate change: config changed from index -1 to 208, VOTER b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195) evicted. New cstate: current_term: 2 leader_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } health_report { overall_health: HEALTHY } } }
I20250624 08:00:18.916870 24791 consensus_queue.cc:237] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 206, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:18.923061 25139 raft_consensus.cc:2953] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 208 to 209, VOTER 55d2201595184d50b969ed31c963640e (127.17.219.194) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } } }
I20250624 08:00:18.935861 24636 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4eb45be406ce41bfba3af333c4cac366 with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 08:00:18.940683 24648 catalog_manager.cc:5582] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d reported cstate change: config changed from index 208 to 209, VOTER 55d2201595184d50b969ed31c963640e (127.17.219.194) evicted. New cstate: current_term: 2 leader_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } health_report { overall_health: HEALTHY } } }
I20250624 08:00:18.998123 25048 tablet_service.cc:1515] Processing DeleteTablet for tablet 4eb45be406ce41bfba3af333c4cac366 with delete_type TABLET_DATA_TOMBSTONED (TS b9544347142c4f3b8e54ad201721bcf7 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:32770
I20250624 08:00:19.000617 24907 tablet_service.cc:1515] Processing DeleteTablet for tablet 4eb45be406ce41bfba3af333c4cac366 with delete_type TABLET_DATA_TOMBSTONED (TS 55d2201595184d50b969ed31c963640e not found in new config with opid_index 209) from {username='slave'} at 127.0.0.1:44196
W20250624 08:00:19.006048 24635 catalog_manager.cc:4908] TS b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485): delete failed for tablet 4eb45be406ce41bfba3af333c4cac366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4eb45be406ce41bfba3af333c4cac366 already in progress: opening tablet
I20250624 08:00:19.012040 25153 tablet_replica.cc:331] stopping tablet replica
I20250624 08:00:19.012914 25153 raft_consensus.cc:2241] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250624 08:00:19.013640 25153 raft_consensus.cc:2270] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 08:00:19.040310 25153 ts_tablet_manager.cc:1905] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 08:00:19.057381 25153 ts_tablet_manager.cc:1918] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.208
I20250624 08:00:19.057904 25153 log.cc:1199] T 4eb45be406ce41bfba3af333c4cac366 P 55d2201595184d50b969ed31c963640e: Deleting WAL directory at /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/wal/wals/4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:19.059736 24635 catalog_manager.cc:4928] TS 55d2201595184d50b969ed31c963640e (127.17.219.194:46855): tablet 4eb45be406ce41bfba3af333c4cac366 (table pre_rebuild [id=f1a812e91f294dbc9580bb060f652d18]) successfully deleted
I20250624 08:00:19.754612 25006 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=206 overwritten=0 applied=206 ignored=0} inserts{seen=10250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:00:19.755815 25006 tablet_bootstrap.cc:492] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Bootstrap complete.
I20250624 08:00:19.757634 25006 ts_tablet_manager.cc:1397] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Time spent bootstrapping tablet: real 2.798s user 2.587s sys 0.065s
I20250624 08:00:19.766978 25006 raft_consensus.cc:357] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:19.769287 25006 raft_consensus.cc:738] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9544347142c4f3b8e54ad201721bcf7, State: Initialized, Role: FOLLOWER
I20250624 08:00:19.770145 25006 consensus_queue.cc:260] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:19.773450 25006 ts_tablet_manager.cc:1428] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20250624 08:00:19.778317 25048 tablet_service.cc:1515] Processing DeleteTablet for tablet 4eb45be406ce41bfba3af333c4cac366 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 209)) from {username='slave'} at 127.0.0.1:32770
I20250624 08:00:19.780900 25175 tablet_replica.cc:331] stopping tablet replica
I20250624 08:00:19.781834 25175 raft_consensus.cc:2241] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 08:00:19.782431 25175 raft_consensus.cc:2270] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 08:00:19.820976 24771 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:00:19.833277 25175 ts_tablet_manager.cc:1905] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 08:00:19.835677 24907 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:00:19.854930 25175 ts_tablet_manager.cc:1918] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.206
I20250624 08:00:19.855482 25175 log.cc:1199] T 4eb45be406ce41bfba3af333c4cac366 P b9544347142c4f3b8e54ad201721bcf7: Deleting WAL directory at /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/wal/wals/4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:19.857388 24635 catalog_manager.cc:4928] TS b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195:41485): tablet 4eb45be406ce41bfba3af333c4cac366 (table pre_rebuild [id=f1a812e91f294dbc9580bb060f652d18]) successfully deleted
I20250624 08:00:19.871207 25048 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
59cafb09591c4e1680cd91853eb1b3b3 | 127.17.219.254:41965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.219.212:39721 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
34ed987cd3d3458f800bfe9b7d1fc94d | 127.17.219.193:40815 | HEALTHY | <none> | 1 | 0
55d2201595184d50b969ed31c963640e | 127.17.219.194:46855 | HEALTHY | <none> | 0 | 0
b9544347142c4f3b8e54ad201721bcf7 | 127.17.219.195:41485 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.17.219.193 | experimental | 127.17.219.193:40815
local_ip_for_outbound_sockets | 127.17.219.194 | experimental | 127.17.219.194:46855
local_ip_for_outbound_sockets | 127.17.219.195 | experimental | 127.17.219.195:41485
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb | hidden | 127.17.219.193:40815
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb | hidden | 127.17.219.194:46855
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb | hidden | 127.17.219.195:41485
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.219.212:39721 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 08:00:20.152477 18287 log_verifier.cc:126] Checking tablet 4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:20.425375 18287 log_verifier.cc:177] Verified matching terms for 209 ops in tablet 4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:20.427685 24648 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56214:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250624 08:00:20.428223 24648 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56214:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250624 08:00:20.440915 24648 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 59cafb09591c4e1680cd91853eb1b3b3: Sending DeleteTablet for 1 replicas of tablet 4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:20.442648 24771 tablet_service.cc:1515] Processing DeleteTablet for tablet 4eb45be406ce41bfba3af333c4cac366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-24 08:00:20 UTC) from {username='slave'} at 127.0.0.1:40988
I20250624 08:00:20.442771 18287 test_util.cc:276] Using random seed: -1203458441
I20250624 08:00:20.444366 25188 tablet_replica.cc:331] stopping tablet replica
I20250624 08:00:20.444936 25188 raft_consensus.cc:2241] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 LEADER]: Raft consensus shutting down.
I20250624 08:00:20.445410 25188 raft_consensus.cc:2270] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 08:00:20.478138 24648 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56252:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 08:00:20.480697 24648 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:00:20.480691 25188 ts_tablet_manager.cc:1905] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 08:00:20.500818 25188 ts_tablet_manager.cc:1918] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.209
I20250624 08:00:20.501276 25188 log.cc:1199] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Deleting WAL directory at /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/wal/wals/4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:20.502218 25188 ts_tablet_manager.cc:1939] T 4eb45be406ce41bfba3af333c4cac366 P 34ed987cd3d3458f800bfe9b7d1fc94d: Deleting consensus metadata
I20250624 08:00:20.504279 24771 tablet_service.cc:1468] Processing CreateTablet for tablet 0ac6260e651e4634ace68a4a81d491df (DEFAULT_TABLE table=post_rebuild [id=347ba9c3bd5c4f21b6cdd450b802080a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:00:20.505199 25048 tablet_service.cc:1468] Processing CreateTablet for tablet 0ac6260e651e4634ace68a4a81d491df (DEFAULT_TABLE table=post_rebuild [id=347ba9c3bd5c4f21b6cdd450b802080a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:00:20.505638 24771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac6260e651e4634ace68a4a81d491df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:00:20.506469 25048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac6260e651e4634ace68a4a81d491df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:00:20.506341 24907 tablet_service.cc:1468] Processing CreateTablet for tablet 0ac6260e651e4634ace68a4a81d491df (DEFAULT_TABLE table=post_rebuild [id=347ba9c3bd5c4f21b6cdd450b802080a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:00:20.507575 24907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac6260e651e4634ace68a4a81d491df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:00:20.511432 24636 catalog_manager.cc:4928] TS 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815): tablet 4eb45be406ce41bfba3af333c4cac366 (table pre_rebuild [id=f1a812e91f294dbc9580bb060f652d18]) successfully deleted
I20250624 08:00:20.526899 25195 tablet_bootstrap.cc:492] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d: Bootstrap starting.
I20250624 08:00:20.528900 25196 tablet_bootstrap.cc:492] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e: Bootstrap starting.
I20250624 08:00:20.530018 25197 tablet_bootstrap.cc:492] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7: Bootstrap starting.
I20250624 08:00:20.535960 25197 tablet_bootstrap.cc:654] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7: Neither blocks nor log segments found. Creating new log.
I20250624 08:00:20.536108 25196 tablet_bootstrap.cc:654] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e: Neither blocks nor log segments found. Creating new log.
I20250624 08:00:20.547097 25197 tablet_bootstrap.cc:492] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7: No bootstrap required, opened a new log
I20250624 08:00:20.547143 25195 tablet_bootstrap.cc:654] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d: Neither blocks nor log segments found. Creating new log.
I20250624 08:00:20.547586 25197 ts_tablet_manager.cc:1397] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20250624 08:00:20.550222 25197 raft_consensus.cc:357] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.551029 25197 raft_consensus.cc:383] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:00:20.551347 25197 raft_consensus.cc:738] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9544347142c4f3b8e54ad201721bcf7, State: Initialized, Role: FOLLOWER
I20250624 08:00:20.553025 25197 consensus_queue.cc:260] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.562054 25197 ts_tablet_manager.cc:1428] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20250624 08:00:20.566695 25195 tablet_bootstrap.cc:492] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d: No bootstrap required, opened a new log
I20250624 08:00:20.567203 25195 ts_tablet_manager.cc:1397] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.020s
I20250624 08:00:20.569799 25195 raft_consensus.cc:357] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.570514 25195 raft_consensus.cc:383] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:00:20.570840 25195 raft_consensus.cc:738] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ed987cd3d3458f800bfe9b7d1fc94d, State: Initialized, Role: FOLLOWER
I20250624 08:00:20.571593 25195 consensus_queue.cc:260] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.573480 25201 raft_consensus.cc:491] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:00:20.573973 25201 raft_consensus.cc:513] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.576167 25196 tablet_bootstrap.cc:492] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e: No bootstrap required, opened a new log
I20250624 08:00:20.576575 25196 ts_tablet_manager.cc:1397] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.004s
I20250624 08:00:20.578521 25201 leader_election.cc:290] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55d2201595184d50b969ed31c963640e (127.17.219.194:46855), 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815)
I20250624 08:00:20.579538 25196 raft_consensus.cc:357] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.590881 25196 raft_consensus.cc:383] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:00:20.591416 25196 raft_consensus.cc:738] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d2201595184d50b969ed31c963640e, State: Initialized, Role: FOLLOWER
I20250624 08:00:20.592347 25196 consensus_queue.cc:260] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.605093 25195 ts_tablet_manager.cc:1428] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d: Time spent starting tablet: real 0.038s user 0.007s sys 0.013s
I20250624 08:00:20.628427 25196 ts_tablet_manager.cc:1428] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e: Time spent starting tablet: real 0.052s user 0.013s sys 0.021s
I20250624 08:00:20.630851 24791 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ac6260e651e4634ace68a4a81d491df" candidate_uuid: "b9544347142c4f3b8e54ad201721bcf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" is_pre_election: true
I20250624 08:00:20.631618 24791 raft_consensus.cc:2466] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9544347142c4f3b8e54ad201721bcf7 in term 0.
I20250624 08:00:20.633119 25003 leader_election.cc:304] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ed987cd3d3458f800bfe9b7d1fc94d, b9544347142c4f3b8e54ad201721bcf7; no voters:
I20250624 08:00:20.634086 25201 raft_consensus.cc:2802] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:00:20.634436 25201 raft_consensus.cc:491] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:00:20.635239 25201 raft_consensus.cc:3058] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:20.642926 25201 raft_consensus.cc:513] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
I20250624 08:00:20.645782 24927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ac6260e651e4634ace68a4a81d491df" candidate_uuid: "b9544347142c4f3b8e54ad201721bcf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d2201595184d50b969ed31c963640e" is_pre_election: true
I20250624 08:00:20.646376 24927 raft_consensus.cc:2466] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9544347142c4f3b8e54ad201721bcf7 in term 0.
I20250624 08:00:20.647447 24926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ac6260e651e4634ace68a4a81d491df" candidate_uuid: "b9544347142c4f3b8e54ad201721bcf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d2201595184d50b969ed31c963640e"
I20250624 08:00:20.647996 24926 raft_consensus.cc:3058] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:20.648366 25201 leader_election.cc:290] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [CANDIDATE]: Term 1 election: Requested vote from peers 55d2201595184d50b969ed31c963640e (127.17.219.194:46855), 34ed987cd3d3458f800bfe9b7d1fc94d (127.17.219.193:40815)
I20250624 08:00:20.649469 24791 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ac6260e651e4634ace68a4a81d491df" candidate_uuid: "b9544347142c4f3b8e54ad201721bcf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d"
I20250624 08:00:20.650023 24791 raft_consensus.cc:3058] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:00:20.655895 24926 raft_consensus.cc:2466] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9544347142c4f3b8e54ad201721bcf7 in term 1.
I20250624 08:00:20.657229 25002 leader_election.cc:304] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d2201595184d50b969ed31c963640e, b9544347142c4f3b8e54ad201721bcf7; no voters:
I20250624 08:00:20.657989 24791 raft_consensus.cc:2466] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9544347142c4f3b8e54ad201721bcf7 in term 1.
I20250624 08:00:20.658124 25201 raft_consensus.cc:2802] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:00:20.667394 25201 raft_consensus.cc:695] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [term 1 LEADER]: Becoming Leader. State: Replica: b9544347142c4f3b8e54ad201721bcf7, State: Running, Role: LEADER
I20250624 08:00:20.668458 25201 consensus_queue.cc:237] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } }
W20250624 08:00:20.677058 24973 tablet.cc:2378] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:00:20.689481 24649 catalog_manager.cc:5582] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9544347142c4f3b8e54ad201721bcf7 (127.17.219.195). New cstate: current_term: 1 leader_uuid: "b9544347142c4f3b8e54ad201721bcf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9544347142c4f3b8e54ad201721bcf7" member_type: VOTER last_known_addr { host: "127.17.219.195" port: 41485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 } health_report { overall_health: UNKNOWN } } }
W20250624 08:00:20.697954 25115 tablet.cc:2378] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 08:00:20.836743 24838 tablet.cc:2378] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:00:20.881801 24926 raft_consensus.cc:1273] T 0ac6260e651e4634ace68a4a81d491df P 55d2201595184d50b969ed31c963640e [term 1 FOLLOWER]: Refusing update from remote peer b9544347142c4f3b8e54ad201721bcf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:00:20.881821 24791 raft_consensus.cc:1273] T 0ac6260e651e4634ace68a4a81d491df P 34ed987cd3d3458f800bfe9b7d1fc94d [term 1 FOLLOWER]: Refusing update from remote peer b9544347142c4f3b8e54ad201721bcf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:00:20.883595 25208 consensus_queue.cc:1035] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d2201595184d50b969ed31c963640e" member_type: VOTER last_known_addr { host: "127.17.219.194" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 08:00:20.884258 25201 consensus_queue.cc:1035] T 0ac6260e651e4634ace68a4a81d491df P b9544347142c4f3b8e54ad201721bcf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ed987cd3d3458f800bfe9b7d1fc94d" member_type: VOTER last_known_addr { host: "127.17.219.193" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:00:20.926954 25217 mvcc.cc:204] Tried to move back new op lower bound from 7171080277515378688 to 7171080276667465728. Current Snapshot: MvccSnapshot[applied={T|T < 7171080277515378688}]
I20250624 08:00:20.937283 25218 mvcc.cc:204] Tried to move back new op lower bound from 7171080277515378688 to 7171080276667465728. Current Snapshot: MvccSnapshot[applied={T|T < 7171080277515378688}]
I20250624 08:00:20.979939 25216 mvcc.cc:204] Tried to move back new op lower bound from 7171080277684690944 to 7171080276667465728. Current Snapshot: MvccSnapshot[applied={T|T < 7171080277617037312}]
I20250624 08:00:26.087670 24907 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:00:26.089512 24771 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:00:26.095587 25048 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250624 08:00:26.295611 24833 debug-util.cc:398] Leaking SignalData structure 0x7b08000b9820 after lost signal to thread 24707
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
59cafb09591c4e1680cd91853eb1b3b3 | 127.17.219.254:41965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.219.212:39721 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
34ed987cd3d3458f800bfe9b7d1fc94d | 127.17.219.193:40815 | HEALTHY | <none> | 0 | 0
55d2201595184d50b969ed31c963640e | 127.17.219.194:46855 | HEALTHY | <none> | 0 | 0
b9544347142c4f3b8e54ad201721bcf7 | 127.17.219.195:41485 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.17.219.193 | experimental | 127.17.219.193:40815
local_ip_for_outbound_sockets | 127.17.219.194 | experimental | 127.17.219.194:46855
local_ip_for_outbound_sockets | 127.17.219.195 | experimental | 127.17.219.195:41485
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-0/data/info.pb | hidden | 127.17.219.193:40815
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-1/data/info.pb | hidden | 127.17.219.194:46855
server_dump_info_path | /tmp/dist-test-taskMO5zxZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750751842019710-18287-0/minicluster-data/ts-2/data/info.pb | hidden | 127.17.219.195:41485
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.219.212:39721 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 08:00:26.335637 18287 log_verifier.cc:126] Checking tablet 0ac6260e651e4634ace68a4a81d491df
I20250624 08:00:27.150355 18287 log_verifier.cc:177] Verified matching terms for 206 ops in tablet 0ac6260e651e4634ace68a4a81d491df
I20250624 08:00:27.151228 18287 log_verifier.cc:126] Checking tablet 4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:27.151520 18287 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 4eb45be406ce41bfba3af333c4cac366
I20250624 08:00:27.176772 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24686
I20250624 08:00:27.220525 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24841
I20250624 08:00:27.261296 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24976
I20250624 08:00:27.306994 18287 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskMO5zxZ/build/tsan/bin/kudu with pid 24616
2025-06-24T08:00:27Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35739 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35739 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (185280 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250624 08:00:27.382063 18287 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 3 messages since previous log ~48 seconds ago