Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250809 07:52:40.366722 849 test_util.cc:276] Using random seed: -108283378
W20250809 07:52:41.493146 849 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.086s user 0.402s sys 0.682s
W20250809 07:52:41.493577 849 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.086s user 0.402s sys 0.682s
I20250809 07:52:41.496104 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:52:41.496337 849 ts_itest-base.cc:116] --------------
I20250809 07:52:41.496531 849 ts_itest-base.cc:117] 4 tablet servers
I20250809 07:52:41.496727 849 ts_itest-base.cc:118] 3 replicas per TS
I20250809 07:52:41.496915 849 ts_itest-base.cc:119] --------------
2025-08-09T07:52:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:52:41Z Disabled control of system clock
I20250809 07:52:41.536612 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:38721
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:38721 with env {}
W20250809 07:52:41.826823 863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:41.827375 863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:52:41.827857 863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:52:41.857856 863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:52:41.858171 863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:52:41.858414 863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:52:41.858641 863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:52:41.895969 863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:38721
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:52:41.897226 863 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:52:41.898766 863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:52:41.910104 869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:41.910264 870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:43.035012 872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:43.037144 871 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1122 milliseconds
I20250809 07:52:43.037236 863 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:52:43.038450 863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:52:43.040931 863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:52:43.042272 863 hybrid_clock.cc:648] HybridClock initialized: now 1754725963042206 us; error 63 us; skew 500 ppm
I20250809 07:52:43.043056 863 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:52:43.049465 863 webserver.cc:489] Webserver started at http://127.0.212.126:46711/ using document root <none> and password file <none>
I20250809 07:52:43.050326 863 fs_manager.cc:362] Metadata directory not provided
I20250809 07:52:43.050576 863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:52:43.051008 863 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:52:43.055153 863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "93d49551af9340e48c69ebdacc52d424"
format_stamp: "Formatted at 2025-08-09 07:52:43 on dist-test-slave-tkp6"
I20250809 07:52:43.056667 863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "93d49551af9340e48c69ebdacc52d424"
format_stamp: "Formatted at 2025-08-09 07:52:43 on dist-test-slave-tkp6"
I20250809 07:52:43.063519 863 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250809 07:52:43.069242 879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:43.070217 863 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250809 07:52:43.070523 863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "93d49551af9340e48c69ebdacc52d424"
format_stamp: "Formatted at 2025-08-09 07:52:43 on dist-test-slave-tkp6"
I20250809 07:52:43.070832 863 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:52:43.125005 863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:52:43.126399 863 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:52:43.126967 863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:52:43.200733 863 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:38721
I20250809 07:52:43.200773 930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:38721 every 8 connection(s)
I20250809 07:52:43.203390 863 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:52:43.208501 931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:52:43.214277 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 863
I20250809 07:52:43.214674 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:52:43.227084 931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Bootstrap starting.
I20250809 07:52:43.232450 931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Neither blocks nor log segments found. Creating new log.
I20250809 07:52:43.234438 931 log.cc:826] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Log is configured to *not* fsync() on all Append() calls
I20250809 07:52:43.239208 931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: No bootstrap required, opened a new log
I20250809 07:52:43.256851 931 raft_consensus.cc:357] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:43.257462 931 raft_consensus.cc:383] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:52:43.257683 931 raft_consensus.cc:738] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d49551af9340e48c69ebdacc52d424, State: Initialized, Role: FOLLOWER
I20250809 07:52:43.258322 931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:43.258811 931 raft_consensus.cc:397] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:52:43.259052 931 raft_consensus.cc:491] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:52:43.259346 931 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:52:43.264009 931 raft_consensus.cc:513] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:43.264653 931 leader_election.cc:304] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d49551af9340e48c69ebdacc52d424; no voters:
I20250809 07:52:43.266548 931 leader_election.cc:290] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:52:43.267135 936 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:52:43.269848 936 raft_consensus.cc:695] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 LEADER]: Becoming Leader. State: Replica: 93d49551af9340e48c69ebdacc52d424, State: Running, Role: LEADER
I20250809 07:52:43.270973 931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:52:43.270632 936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:43.282255 937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d49551af9340e48c69ebdacc52d424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } } }
I20250809 07:52:43.282263 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d49551af9340e48c69ebdacc52d424. Latest consensus state: current_term: 1 leader_uuid: "93d49551af9340e48c69ebdacc52d424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } } }
I20250809 07:52:43.282949 937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: This master's current role is: LEADER
I20250809 07:52:43.282948 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: This master's current role is: LEADER
I20250809 07:52:43.288618 944 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:52:43.300994 944 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:52:43.319553 944 catalog_manager.cc:1349] Generated new cluster ID: d0521ffcf68f48f08e5502a65742d561
I20250809 07:52:43.319813 944 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:52:43.341223 944 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:52:43.343104 944 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:52:43.369930 944 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Generated new TSK 0
I20250809 07:52:43.371039 944 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:52:43.382570 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--builtin_ntp_servers=127.0.212.84:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 07:52:43.681917 955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:43.682392 955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:52:43.682860 955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:52:43.713418 955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:52:43.714244 955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:52:43.748163 955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:52:43.749419 955 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:52:43.751066 955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:52:43.764007 961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:43.765905 962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:45.127111 964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:45.130062 963 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1360 milliseconds
W20250809 07:52:45.131003 955 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.367s user 0.522s sys 0.837s
W20250809 07:52:45.131253 955 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.367s user 0.522s sys 0.837s
I20250809 07:52:45.131464 955 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:52:45.132465 955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:52:45.136344 955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:52:45.137779 955 hybrid_clock.cc:648] HybridClock initialized: now 1754725965137749 us; error 44 us; skew 500 ppm
I20250809 07:52:45.138579 955 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:52:45.148075 955 webserver.cc:489] Webserver started at http://127.0.212.65:36009/ using document root <none> and password file <none>
I20250809 07:52:45.149024 955 fs_manager.cc:362] Metadata directory not provided
I20250809 07:52:45.149251 955 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:52:45.149717 955 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:52:45.153880 955 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f7857c0655ee471d8cadd2de6452a687"
format_stamp: "Formatted at 2025-08-09 07:52:45 on dist-test-slave-tkp6"
I20250809 07:52:45.154938 955 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f7857c0655ee471d8cadd2de6452a687"
format_stamp: "Formatted at 2025-08-09 07:52:45 on dist-test-slave-tkp6"
I20250809 07:52:45.165382 955 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.004s sys 0.008s
I20250809 07:52:45.172075 971 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:45.173099 955 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250809 07:52:45.173388 955 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f7857c0655ee471d8cadd2de6452a687"
format_stamp: "Formatted at 2025-08-09 07:52:45 on dist-test-slave-tkp6"
I20250809 07:52:45.173688 955 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:52:45.238773 955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:52:45.240242 955 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:52:45.240695 955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:52:45.243490 955 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:52:45.248199 955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:52:45.248401 955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:45.248651 955 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:52:45.248805 955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:45.416100 955 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:41283
I20250809 07:52:45.416179 1083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:41283 every 8 connection(s)
I20250809 07:52:45.418556 955 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:52:45.425599 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 955
I20250809 07:52:45.425971 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:52:45.433017 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--builtin_ntp_servers=127.0.212.84:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:52:45.445991 1084 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:45.446406 1084 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:45.447866 1084 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:45.451081 896 ts_manager.cc:194] Registered new tserver with Master: f7857c0655ee471d8cadd2de6452a687 (127.0.212.65:41283)
I20250809 07:52:45.453275 896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:54915
W20250809 07:52:45.751506 1088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:45.752007 1088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:52:45.752460 1088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:52:45.782740 1088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:52:45.783532 1088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:52:45.817960 1088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:52:45.819142 1088 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:52:45.820710 1088 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:52:45.833221 1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:52:46.456730 1084 heartbeater.cc:499] Master 127.0.212.126:38721 was elected leader, sending a full tablet report...
W20250809 07:52:45.834452 1095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:46.936038 1096 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1099 milliseconds
W20250809 07:52:46.936290 1097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:46.939786 1088 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.106s user 0.371s sys 0.735s
W20250809 07:52:46.940119 1088 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.107s user 0.371s sys 0.735s
I20250809 07:52:46.940377 1088 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:52:46.941722 1088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:52:46.944248 1088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:52:46.945742 1088 hybrid_clock.cc:648] HybridClock initialized: now 1754725966945680 us; error 39 us; skew 500 ppm
I20250809 07:52:46.946785 1088 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:52:46.954970 1088 webserver.cc:489] Webserver started at http://127.0.212.66:44261/ using document root <none> and password file <none>
I20250809 07:52:46.955994 1088 fs_manager.cc:362] Metadata directory not provided
I20250809 07:52:46.956207 1088 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:52:46.956627 1088 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:52:46.960737 1088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "23e69886651c47739e37968aac76d176"
format_stamp: "Formatted at 2025-08-09 07:52:46 on dist-test-slave-tkp6"
I20250809 07:52:46.961812 1088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "23e69886651c47739e37968aac76d176"
format_stamp: "Formatted at 2025-08-09 07:52:46 on dist-test-slave-tkp6"
I20250809 07:52:46.969400 1088 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 07:52:46.975453 1104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:46.976547 1088 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250809 07:52:46.976852 1088 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "23e69886651c47739e37968aac76d176"
format_stamp: "Formatted at 2025-08-09 07:52:46 on dist-test-slave-tkp6"
I20250809 07:52:46.977185 1088 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:52:47.037038 1088 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:52:47.038447 1088 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:52:47.038861 1088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:52:47.041616 1088 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:52:47.045887 1088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:52:47.046089 1088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:47.046329 1088 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:52:47.046487 1088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:47.189190 1088 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:40871
I20250809 07:52:47.189284 1216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:40871 every 8 connection(s)
I20250809 07:52:47.191798 1088 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:52:47.196449 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1088
I20250809 07:52:47.196909 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 07:52:47.204165 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--builtin_ntp_servers=127.0.212.84:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:52:47.223230 1217 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:47.223656 1217 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:47.225145 1217 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:47.227137 896 ts_manager.cc:194] Registered new tserver with Master: 23e69886651c47739e37968aac76d176 (127.0.212.66:40871)
I20250809 07:52:47.228335 896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:43083
W20250809 07:52:47.497412 1221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:47.497884 1221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:52:47.498406 1221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:52:47.528936 1221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:52:47.529814 1221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:52:47.563361 1221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:52:47.564687 1221 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:52:47.566169 1221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:52:47.585266 1227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:52:48.231187 1217 heartbeater.cc:499] Master 127.0.212.126:38721 was elected leader, sending a full tablet report...
W20250809 07:52:47.585882 1228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:48.780544 1230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:48.782164 1229 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1197 milliseconds
W20250809 07:52:48.783181 1221 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.200s user 0.502s sys 0.687s
W20250809 07:52:48.783421 1221 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.200s user 0.502s sys 0.687s
I20250809 07:52:48.783629 1221 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:52:48.784641 1221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:52:48.787117 1221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:52:48.788491 1221 hybrid_clock.cc:648] HybridClock initialized: now 1754725968788432 us; error 55 us; skew 500 ppm
I20250809 07:52:48.789285 1221 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:52:48.797288 1221 webserver.cc:489] Webserver started at http://127.0.212.67:37527/ using document root <none> and password file <none>
I20250809 07:52:48.798233 1221 fs_manager.cc:362] Metadata directory not provided
I20250809 07:52:48.798436 1221 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:52:48.798859 1221 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:52:48.803105 1221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bd2e4982ad0246ee86f189299d990e77"
format_stamp: "Formatted at 2025-08-09 07:52:48 on dist-test-slave-tkp6"
I20250809 07:52:48.804168 1221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bd2e4982ad0246ee86f189299d990e77"
format_stamp: "Formatted at 2025-08-09 07:52:48 on dist-test-slave-tkp6"
I20250809 07:52:48.811995 1221 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250809 07:52:48.817880 1237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:48.818964 1221 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250809 07:52:48.819248 1221 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bd2e4982ad0246ee86f189299d990e77"
format_stamp: "Formatted at 2025-08-09 07:52:48 on dist-test-slave-tkp6"
I20250809 07:52:48.819550 1221 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:52:48.907308 1221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:52:48.908763 1221 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:52:48.909176 1221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:52:48.912032 1221 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:52:48.915835 1221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:52:48.916042 1221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:48.916260 1221 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:52:48.916410 1221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:49.057339 1221 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:42555
I20250809 07:52:49.057416 1349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:42555 every 8 connection(s)
I20250809 07:52:49.059772 1221 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:52:49.069572 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1221
I20250809 07:52:49.070034 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 07:52:49.076464 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.68:0
--local_ip_for_outbound_sockets=127.0.212.68
--webserver_interface=127.0.212.68
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--builtin_ntp_servers=127.0.212.84:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:52:49.082933 1350 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:49.083345 1350 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:49.084537 1350 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:49.086946 896 ts_manager.cc:194] Registered new tserver with Master: bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
I20250809 07:52:49.088208 896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:41407
W20250809 07:52:49.375558 1354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:49.376055 1354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:52:49.376539 1354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:52:49.406790 1354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:52:49.407881 1354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.68
I20250809 07:52:49.442193 1354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.212.68
--webserver_port=0
--tserver_master_addrs=127.0.212.126:38721
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.68
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:52:49.443388 1354 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:52:49.444859 1354 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:52:49.457753 1360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:52:50.091717 1350 heartbeater.cc:499] Master 127.0.212.126:38721 was elected leader, sending a full tablet report...
W20250809 07:52:49.458182 1361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:50.580585 1363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:50.582274 1362 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1119 milliseconds
I20250809 07:52:50.582405 1354 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:52:50.583566 1354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:52:50.585613 1354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:52:50.586958 1354 hybrid_clock.cc:648] HybridClock initialized: now 1754725970586918 us; error 47 us; skew 500 ppm
I20250809 07:52:50.587762 1354 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:52:50.593880 1354 webserver.cc:489] Webserver started at http://127.0.212.68:32995/ using document root <none> and password file <none>
I20250809 07:52:50.594733 1354 fs_manager.cc:362] Metadata directory not provided
I20250809 07:52:50.594931 1354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:52:50.595371 1354 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:52:50.599586 1354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6b397062c5a646189e8cb40152ecf781"
format_stamp: "Formatted at 2025-08-09 07:52:50 on dist-test-slave-tkp6"
I20250809 07:52:50.600641 1354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6b397062c5a646189e8cb40152ecf781"
format_stamp: "Formatted at 2025-08-09 07:52:50 on dist-test-slave-tkp6"
I20250809 07:52:50.607913 1354 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250809 07:52:50.613052 1371 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:50.613996 1354 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250809 07:52:50.614284 1354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6b397062c5a646189e8cb40152ecf781"
format_stamp: "Formatted at 2025-08-09 07:52:50 on dist-test-slave-tkp6"
I20250809 07:52:50.614594 1354 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:52:50.675951 1354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:52:50.677377 1354 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:52:50.678252 1354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:52:50.680531 1354 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:52:50.684543 1354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:52:50.684759 1354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:50.685024 1354 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:52:50.685180 1354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:50.826159 1354 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.68:43721
I20250809 07:52:50.826247 1483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.68:43721 every 8 connection(s)
I20250809 07:52:50.828619 1354 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250809 07:52:50.838287 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1354
I20250809 07:52:50.838800 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250809 07:52:50.849653 1484 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:50.850212 1484 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:50.851200 1484 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:50.853539 896 ts_manager.cc:194] Registered new tserver with Master: 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721)
I20250809 07:52:50.854696 896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.68:56947
I20250809 07:52:50.859171 849 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250809 07:52:50.899433 896 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48162:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250809 07:52:50.969463 1285 tablet_service.cc:1468] Processing CreateTablet for tablet f4dd206347b14c12a5e03bc206abdf47 (DEFAULT_TABLE table=TestTable [id=31d7bb28f9214c57a036bb231dadf9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:52:50.970216 1419 tablet_service.cc:1468] Processing CreateTablet for tablet f4dd206347b14c12a5e03bc206abdf47 (DEFAULT_TABLE table=TestTable [id=31d7bb28f9214c57a036bb231dadf9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:52:50.971233 1285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dd206347b14c12a5e03bc206abdf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:52:50.971691 1419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dd206347b14c12a5e03bc206abdf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:52:50.971629 1152 tablet_service.cc:1468] Processing CreateTablet for tablet f4dd206347b14c12a5e03bc206abdf47 (DEFAULT_TABLE table=TestTable [id=31d7bb28f9214c57a036bb231dadf9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:52:50.973323 1152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dd206347b14c12a5e03bc206abdf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:52:50.995592 1503 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: Bootstrap starting.
I20250809 07:52:50.998194 1504 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Bootstrap starting.
I20250809 07:52:51.000484 1505 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: Bootstrap starting.
I20250809 07:52:51.006021 1505 tablet_bootstrap.cc:654] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: Neither blocks nor log segments found. Creating new log.
I20250809 07:52:51.006094 1504 tablet_bootstrap.cc:654] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Neither blocks nor log segments found. Creating new log.
I20250809 07:52:51.006671 1503 tablet_bootstrap.cc:654] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: Neither blocks nor log segments found. Creating new log.
I20250809 07:52:51.007781 1505 log.cc:826] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: Log is configured to *not* fsync() on all Append() calls
I20250809 07:52:51.008692 1504 log.cc:826] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Log is configured to *not* fsync() on all Append() calls
I20250809 07:52:51.011098 1503 log.cc:826] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: Log is configured to *not* fsync() on all Append() calls
I20250809 07:52:51.013155 1505 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: No bootstrap required, opened a new log
I20250809 07:52:51.013567 1505 ts_tablet_manager.cc:1397] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250809 07:52:51.015349 1504 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: No bootstrap required, opened a new log
I20250809 07:52:51.015928 1504 ts_tablet_manager.cc:1397] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20250809 07:52:51.022912 1503 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: No bootstrap required, opened a new log
I20250809 07:52:51.023521 1503 ts_tablet_manager.cc:1397] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.004s
I20250809 07:52:51.036351 1504 raft_consensus.cc:357] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.036909 1505 raft_consensus.cc:357] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.037532 1504 raft_consensus.cc:383] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:52:51.037799 1504 raft_consensus.cc:738] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2e4982ad0246ee86f189299d990e77, State: Initialized, Role: FOLLOWER
I20250809 07:52:51.037804 1505 raft_consensus.cc:383] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:52:51.038111 1505 raft_consensus.cc:738] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e69886651c47739e37968aac76d176, State: Initialized, Role: FOLLOWER
I20250809 07:52:51.038745 1504 consensus_queue.cc:260] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.038910 1505 consensus_queue.cc:260] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.047360 1505 ts_tablet_manager.cc:1428] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: Time spent starting tablet: real 0.034s user 0.033s sys 0.000s
I20250809 07:52:51.047439 1504 ts_tablet_manager.cc:1428] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Time spent starting tablet: real 0.031s user 0.021s sys 0.008s
I20250809 07:52:51.050444 1503 raft_consensus.cc:357] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.051174 1503 raft_consensus.cc:383] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:52:51.051434 1503 raft_consensus.cc:738] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b397062c5a646189e8cb40152ecf781, State: Initialized, Role: FOLLOWER
I20250809 07:52:51.052172 1503 consensus_queue.cc:260] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.055928 1484 heartbeater.cc:499] Master 127.0.212.126:38721 was elected leader, sending a full tablet report...
I20250809 07:52:51.057055 1503 ts_tablet_manager.cc:1428] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: Time spent starting tablet: real 0.033s user 0.032s sys 0.000s
W20250809 07:52:51.066056 1351 tablet.cc:2378] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 07:52:51.083570 1485 tablet.cc:2378] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 07:52:51.200234 1218 tablet.cc:2378] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:52:51.227509 1509 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:52:51.228125 1509 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.233572 1509 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721), bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
I20250809 07:52:51.244778 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "23e69886651c47739e37968aac76d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2e4982ad0246ee86f189299d990e77" is_pre_election: true
I20250809 07:52:51.245075 1439 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "23e69886651c47739e37968aac76d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b397062c5a646189e8cb40152ecf781" is_pre_election: true
I20250809 07:52:51.245658 1305 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23e69886651c47739e37968aac76d176 in term 0.
I20250809 07:52:51.245831 1439 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23e69886651c47739e37968aac76d176 in term 0.
I20250809 07:52:51.247382 1105 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23e69886651c47739e37968aac76d176, bd2e4982ad0246ee86f189299d990e77; no voters:
I20250809 07:52:51.248122 1509 raft_consensus.cc:2802] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:52:51.248409 1509 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:52:51.248692 1509 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:52:51.253472 1509 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.254762 1509 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [CANDIDATE]: Term 1 election: Requested vote from peers 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721), bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
I20250809 07:52:51.255486 1439 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "23e69886651c47739e37968aac76d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b397062c5a646189e8cb40152ecf781"
I20250809 07:52:51.255575 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "23e69886651c47739e37968aac76d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2e4982ad0246ee86f189299d990e77"
I20250809 07:52:51.255903 1439 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:52:51.256103 1305 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:52:51.260495 1439 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23e69886651c47739e37968aac76d176 in term 1.
I20250809 07:52:51.260789 1305 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23e69886651c47739e37968aac76d176 in term 1.
I20250809 07:52:51.261617 1108 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23e69886651c47739e37968aac76d176, 6b397062c5a646189e8cb40152ecf781; no voters:
I20250809 07:52:51.262316 1509 raft_consensus.cc:2802] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:52:51.263775 1509 raft_consensus.cc:695] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [term 1 LEADER]: Becoming Leader. State: Replica: 23e69886651c47739e37968aac76d176, State: Running, Role: LEADER
I20250809 07:52:51.265138 1509 consensus_queue.cc:237] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:51.274021 896 catalog_manager.cc:5582] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23e69886651c47739e37968aac76d176 (127.0.212.66). New cstate: current_term: 1 leader_uuid: "23e69886651c47739e37968aac76d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } health_report { overall_health: UNKNOWN } } }
I20250809 07:52:51.343899 849 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250809 07:52:51.347384 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 23e69886651c47739e37968aac76d176 to finish bootstrapping
I20250809 07:52:51.360777 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bd2e4982ad0246ee86f189299d990e77 to finish bootstrapping
I20250809 07:52:51.371016 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6b397062c5a646189e8cb40152ecf781 to finish bootstrapping
I20250809 07:52:51.381969 849 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250809 07:52:51.385077 849 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "f4dd206347b14c12a5e03bc206abdf47"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "6b397062c5a646189e8cb40152ecf781"
rpc_addresses {
host: "127.0.212.68"
port: 43721
}
}
ts_infos {
permanent_uuid: "23e69886651c47739e37968aac76d176"
rpc_addresses {
host: "127.0.212.66"
port: 40871
}
}
ts_infos {
permanent_uuid: "bd2e4982ad0246ee86f189299d990e77"
rpc_addresses {
host: "127.0.212.67"
port: 42555
}
}
I20250809 07:52:51.672369 1509 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:52:51.687142 1509 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P 23e69886651c47739e37968aac76d176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:52:51.689872 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1088
W20250809 07:52:51.715474 1239 connection.cc:537] server connection from 127.0.212.66:50701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250809 07:52:51.715498 1375 connection.cc:537] server connection from 127.0.212.66:58295 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250809 07:52:51.716148 882 connection.cc:537] server connection from 127.0.212.66:43083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250809 07:52:51.717139 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 863
I20250809 07:52:51.743592 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:38721
--webserver_interface=127.0.212.126
--webserver_port=46711
--builtin_ntp_servers=127.0.212.84:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:38721 with env {}
W20250809 07:52:52.047271 1525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:52.047859 1525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:52:52.048292 1525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:52:52.078795 1525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:52:52.079095 1525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:52:52.079335 1525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:52:52.079566 1525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:52:52.113799 1525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:38721
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=46711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:52:52.115000 1525 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:52:52.116521 1525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:52:52.126243 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:52.483377 1084 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:38721 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:38721: connect: Connection refused (error 111)
W20250809 07:52:52.703529 1484 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:38721 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:38721: connect: Connection refused (error 111)
W20250809 07:52:52.712255 1350 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:38721 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:38721: connect: Connection refused (error 111)
I20250809 07:52:53.185479 1540 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23e69886651c47739e37968aac76d176)
I20250809 07:52:53.185868 1540 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:53.188038 1540 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23e69886651c47739e37968aac76d176 (127.0.212.66:40871), bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
W20250809 07:52:53.189559 1373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.212.66:40871: connect: Connection refused (error 111)
I20250809 07:52:53.205329 1544 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23e69886651c47739e37968aac76d176)
W20250809 07:52:53.205461 1373 leader_election.cc:336] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 23e69886651c47739e37968aac76d176 (127.0.212.66:40871): Network error: Client connection negotiation failed: client connection to 127.0.212.66:40871: connect: Connection refused (error 111)
I20250809 07:52:53.205852 1544 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:53.210034 1544 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721), 23e69886651c47739e37968aac76d176 (127.0.212.66:40871)
W20250809 07:52:53.211644 1239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.212.66:40871: connect: Connection refused (error 111)
I20250809 07:52:53.212961 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "6b397062c5a646189e8cb40152ecf781" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd2e4982ad0246ee86f189299d990e77" is_pre_election: true
I20250809 07:52:53.213512 1305 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b397062c5a646189e8cb40152ecf781 in term 1.
I20250809 07:52:53.214708 1372 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b397062c5a646189e8cb40152ecf781, bd2e4982ad0246ee86f189299d990e77; no voters: 23e69886651c47739e37968aac76d176
I20250809 07:52:53.215608 1540 raft_consensus.cc:2802] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250809 07:52:53.215986 1540 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 23e69886651c47739e37968aac76d176)
I20250809 07:52:53.216307 1540 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 1 FOLLOWER]: Advancing to term 2
W20250809 07:52:53.219000 1239 leader_election.cc:336] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 23e69886651c47739e37968aac76d176 (127.0.212.66:40871): Network error: Client connection negotiation failed: client connection to 127.0.212.66:40871: connect: Connection refused (error 111)
I20250809 07:52:53.222257 1540 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:53.223592 1439 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "bd2e4982ad0246ee86f189299d990e77" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b397062c5a646189e8cb40152ecf781" is_pre_election: true
I20250809 07:52:53.224329 1439 raft_consensus.cc:2391] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd2e4982ad0246ee86f189299d990e77 in current term 2: Already voted for candidate 6b397062c5a646189e8cb40152ecf781 in this term.
I20250809 07:52:53.225252 1305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "6b397062c5a646189e8cb40152ecf781" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd2e4982ad0246ee86f189299d990e77"
I20250809 07:52:53.225764 1305 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:52:53.225641 1241 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd2e4982ad0246ee86f189299d990e77; no voters: 23e69886651c47739e37968aac76d176, 6b397062c5a646189e8cb40152ecf781
I20250809 07:52:53.226976 1540 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [CANDIDATE]: Term 2 election: Requested vote from peers 23e69886651c47739e37968aac76d176 (127.0.212.66:40871), bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
W20250809 07:52:53.227577 1373 leader_election.cc:336] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 23e69886651c47739e37968aac76d176 (127.0.212.66:40871): Network error: Client connection negotiation failed: client connection to 127.0.212.66:40871: connect: Connection refused (error 111)
I20250809 07:52:53.231750 1305 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b397062c5a646189e8cb40152ecf781 in term 2.
I20250809 07:52:53.232223 1544 raft_consensus.cc:2747] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250809 07:52:53.232779 1372 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b397062c5a646189e8cb40152ecf781, bd2e4982ad0246ee86f189299d990e77; no voters: 23e69886651c47739e37968aac76d176
I20250809 07:52:53.233330 1540 raft_consensus.cc:2802] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:52:53.235064 1540 raft_consensus.cc:695] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 LEADER]: Becoming Leader. State: Replica: 6b397062c5a646189e8cb40152ecf781, State: Running, Role: LEADER
I20250809 07:52:53.235929 1540 consensus_queue.cc:237] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
W20250809 07:52:52.127521 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:52.130697 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:52:53.243217 1533 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 07:52:53.243250 1525 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:52:53.247190 1525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:52:53.249616 1525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:52:53.250970 1525 hybrid_clock.cc:648] HybridClock initialized: now 1754725973250935 us; error 40 us; skew 500 ppm
I20250809 07:52:53.251775 1525 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:52:53.257544 1525 webserver.cc:489] Webserver started at http://127.0.212.126:46711/ using document root <none> and password file <none>
I20250809 07:52:53.258437 1525 fs_manager.cc:362] Metadata directory not provided
I20250809 07:52:53.258661 1525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:52:53.266216 1525 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250809 07:52:53.272310 1554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:52:53.273316 1525 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250809 07:52:53.273627 1525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "93d49551af9340e48c69ebdacc52d424"
format_stamp: "Formatted at 2025-08-09 07:52:43 on dist-test-slave-tkp6"
I20250809 07:52:53.275506 1525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:52:53.328045 1525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:52:53.329450 1525 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:52:53.329885 1525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:52:53.395442 1525 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:38721
I20250809 07:52:53.395493 1605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:38721 every 8 connection(s)
I20250809 07:52:53.398131 1525 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:52:53.403900 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1525
I20250809 07:52:53.404381 849 kudu-admin-test.cc:735] Forcing unsafe config change on tserver bd2e4982ad0246ee86f189299d990e77
I20250809 07:52:53.407630 1606 sys_catalog.cc:263] Verifying existing consensus state
I20250809 07:52:53.412663 1606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Bootstrap starting.
I20250809 07:52:53.461789 1606 log.cc:826] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Log is configured to *not* fsync() on all Append() calls
I20250809 07:52:53.492163 1606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:52:53.493531 1606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Bootstrap complete.
I20250809 07:52:53.520195 1606 raft_consensus.cc:357] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:53.522320 1606 raft_consensus.cc:738] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d49551af9340e48c69ebdacc52d424, State: Initialized, Role: FOLLOWER
I20250809 07:52:53.523078 1606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:53.523559 1606 raft_consensus.cc:397] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:52:53.523854 1606 raft_consensus.cc:491] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:52:53.524168 1606 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:52:53.530603 1606 raft_consensus.cc:513] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:53.531476 1606 leader_election.cc:304] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d49551af9340e48c69ebdacc52d424; no voters:
I20250809 07:52:53.533136 1084 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:53.533612 1606 leader_election.cc:290] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 07:52:53.533919 1612 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:52:53.541859 1612 raft_consensus.cc:695] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [term 2 LEADER]: Becoming Leader. State: Replica: 93d49551af9340e48c69ebdacc52d424, State: Running, Role: LEADER
I20250809 07:52:53.542668 1606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:52:53.542673 1612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } }
I20250809 07:52:53.552487 1613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93d49551af9340e48c69ebdacc52d424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } } }
I20250809 07:52:53.552541 1614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d49551af9340e48c69ebdacc52d424. Latest consensus state: current_term: 2 leader_uuid: "93d49551af9340e48c69ebdacc52d424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d49551af9340e48c69ebdacc52d424" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 38721 } } }
I20250809 07:52:53.553311 1613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: This master's current role is: LEADER
I20250809 07:52:53.553334 1614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424 [sys.catalog]: This master's current role is: LEADER
I20250809 07:52:53.565150 1619 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:52:53.581233 1619 catalog_manager.cc:671] Loaded metadata for table TestTable [id=31d7bb28f9214c57a036bb231dadf9e6]
I20250809 07:52:53.588745 1619 tablet_loader.cc:96] loaded metadata for tablet f4dd206347b14c12a5e03bc206abdf47 (table TestTable [id=31d7bb28f9214c57a036bb231dadf9e6])
I20250809 07:52:53.591341 1619 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:52:53.595810 1619 catalog_manager.cc:1261] Loaded cluster ID: d0521ffcf68f48f08e5502a65742d561
I20250809 07:52:53.596164 1619 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:52:53.603147 1619 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:52:53.608621 1619 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 93d49551af9340e48c69ebdacc52d424: Loaded TSK: 0
I20250809 07:52:53.610164 1619 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250809 07:52:53.695111 1373 consensus_peers.cc:489] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 -> Peer 23e69886651c47739e37968aac76d176 (127.0.212.66:40871): Couldn't send request to peer 23e69886651c47739e37968aac76d176. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.66:40871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250809 07:52:53.724073 1350 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:53.730294 1571 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" instance_seqno: 1754725969025346) as {username='slave'} at 127.0.212.67:60571; Asking this server to re-register.
I20250809 07:52:53.731956 1350 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:53.732616 1350 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:53.736296 1571 ts_manager.cc:194] Registered new tserver with Master: bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
W20250809 07:52:53.768085 1608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:52:53.768692 1608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250809 07:52:53.771068 1305 raft_consensus.cc:1273] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 2 FOLLOWER]: Refusing update from remote peer 6b397062c5a646189e8cb40152ecf781: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250809 07:52:53.772500 1540 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250809 07:52:53.780673 1571 catalog_manager.cc:5582] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 reported cstate change: term changed from 1 to 2, leader changed from 23e69886651c47739e37968aac76d176 (127.0.212.66) to 6b397062c5a646189e8cb40152ecf781 (127.0.212.68). New cstate: current_term: 2 leader_uuid: "6b397062c5a646189e8cb40152ecf781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } }
W20250809 07:52:53.820261 1608 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250809 07:52:53.820272 1484 heartbeater.cc:344] Connected to a master server at 127.0.212.126:38721
I20250809 07:52:53.838646 1570 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6b397062c5a646189e8cb40152ecf781" instance_seqno: 1754725970792376) as {username='slave'} at 127.0.212.68:37529; Asking this server to re-register.
I20250809 07:52:53.840162 1484 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:53.840781 1484 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:53.844053 1571 ts_manager.cc:194] Registered new tserver with Master: 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721)
I20250809 07:52:54.539147 1570 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" instance_seqno: 1754725965377908) as {username='slave'} at 127.0.212.65:50007; Asking this server to re-register.
I20250809 07:52:54.541107 1084 heartbeater.cc:461] Registering TS with master...
I20250809 07:52:54.541824 1084 heartbeater.cc:507] Master 127.0.212.126:38721 requested a full tablet report, sending...
I20250809 07:52:54.544059 1570 ts_manager.cc:194] Registered new tserver with Master: f7857c0655ee471d8cadd2de6452a687 (127.0.212.65:41283)
W20250809 07:52:55.274453 1641 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 1608
W20250809 07:52:55.274962 1641 kernel_stack_watchdog.cc:198] Thread 1608 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 07:52:55.333776 1608 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.462s user 0.517s sys 0.908s
W20250809 07:52:55.450507 1608 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.579s user 0.517s sys 0.920s
I20250809 07:52:55.497314 1305 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "bd2e4982ad0246ee86f189299d990e77"
tablet_id: "f4dd206347b14c12a5e03bc206abdf47"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "6b397062c5a646189e8cb40152ecf781"
}
peers {
permanent_uuid: "bd2e4982ad0246ee86f189299d990e77"
}
}
from {username='slave'} at 127.0.0.1:55230
W20250809 07:52:55.498306 1305 raft_consensus.cc:2216] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true
I20250809 07:52:55.499079 1305 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:52:55.781198 1305 raft_consensus.cc:1238] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 3 FOLLOWER]: Rejecting Update request from peer 6b397062c5a646189e8cb40152ecf781 for earlier term 2. Current term is 3. Ops: []
I20250809 07:52:55.782179 1647 consensus_queue.cc:1046] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250809 07:52:55.783226 1647 raft_consensus.cc:3053] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 LEADER]: Stepping down as leader of term 2
I20250809 07:52:55.783447 1647 raft_consensus.cc:738] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b397062c5a646189e8cb40152ecf781, State: Running, Role: LEADER
I20250809 07:52:55.784031 1647 consensus_queue.cc:260] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } }
I20250809 07:52:55.784868 1647 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 2 FOLLOWER]: Advancing to term 3
W20250809 07:52:55.848297 1480 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc700 after lost signal to thread 1355
W20250809 07:52:55.849025 1480 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7120 after lost signal to thread 1483
I20250809 07:52:57.005388 1665 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250809 07:52:57.005752 1665 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true
I20250809 07:52:57.006763 1665 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721)
I20250809 07:52:57.007992 1439 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "bd2e4982ad0246ee86f189299d990e77" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b397062c5a646189e8cb40152ecf781" is_pre_election: true
I20250809 07:52:57.008623 1439 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd2e4982ad0246ee86f189299d990e77 in term 3.
I20250809 07:52:57.009635 1241 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6b397062c5a646189e8cb40152ecf781, bd2e4982ad0246ee86f189299d990e77; no voters:
I20250809 07:52:57.010308 1665 raft_consensus.cc:2802] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250809 07:52:57.010586 1665 raft_consensus.cc:491] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250809 07:52:57.010828 1665 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 3 FOLLOWER]: Advancing to term 4
I20250809 07:52:57.014813 1665 raft_consensus.cc:513] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true
I20250809 07:52:57.015815 1665 leader_election.cc:290] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 4 election: Requested vote from peers 6b397062c5a646189e8cb40152ecf781 (127.0.212.68:43721)
I20250809 07:52:57.016808 1439 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dd206347b14c12a5e03bc206abdf47" candidate_uuid: "bd2e4982ad0246ee86f189299d990e77" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b397062c5a646189e8cb40152ecf781"
I20250809 07:52:57.017344 1439 raft_consensus.cc:3058] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 3 FOLLOWER]: Advancing to term 4
I20250809 07:52:57.023455 1439 raft_consensus.cc:2466] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2e4982ad0246ee86f189299d990e77 in term 4.
I20250809 07:52:57.024420 1241 leader_election.cc:304] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6b397062c5a646189e8cb40152ecf781, bd2e4982ad0246ee86f189299d990e77; no voters:
I20250809 07:52:57.025192 1665 raft_consensus.cc:2802] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 4 FOLLOWER]: Leader election won for term 4
I20250809 07:52:57.026209 1665 raft_consensus.cc:695] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 4 LEADER]: Becoming Leader. State: Replica: bd2e4982ad0246ee86f189299d990e77, State: Running, Role: LEADER
I20250809 07:52:57.026927 1665 consensus_queue.cc:237] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true
I20250809 07:52:57.033751 1570 catalog_manager.cc:5582] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 reported cstate change: term changed from 2 to 4, leader changed from 6b397062c5a646189e8cb40152ecf781 (127.0.212.68) to bd2e4982ad0246ee86f189299d990e77 (127.0.212.67), now has a pending config: VOTER 6b397062c5a646189e8cb40152ecf781 (127.0.212.68), VOTER bd2e4982ad0246ee86f189299d990e77 (127.0.212.67). New cstate: current_term: 4 leader_uuid: "bd2e4982ad0246ee86f189299d990e77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23e69886651c47739e37968aac76d176" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40871 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true }
I20250809 07:52:57.636999 1439 raft_consensus.cc:1273] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Refusing update from remote peer bd2e4982ad0246ee86f189299d990e77: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250809 07:52:57.638382 1673 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250809 07:52:57.646287 1676 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 23e69886651c47739e37968aac76d176 (127.0.212.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true }
I20250809 07:52:57.647142 1439 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 23e69886651c47739e37968aac76d176 (127.0.212.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } unsafe_config_change: true }
I20250809 07:52:57.657497 1570 catalog_manager.cc:5582] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 reported cstate change: config changed from index -1 to 3, VOTER 23e69886651c47739e37968aac76d176 (127.0.212.66) evicted, no longer has a pending config: VOTER 6b397062c5a646189e8cb40152ecf781 (127.0.212.68), VOTER bd2e4982ad0246ee86f189299d990e77 (127.0.212.67). New cstate: current_term: 4 leader_uuid: "bd2e4982ad0246ee86f189299d990e77" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250809 07:52:57.665486 1570 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet f4dd206347b14c12a5e03bc206abdf47 on TS 23e69886651c47739e37968aac76d176: Not found: failed to reset TS proxy: Could not find TS for UUID 23e69886651c47739e37968aac76d176
I20250809 07:52:57.681138 1305 consensus_queue.cc:237] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: true } } unsafe_config_change: true
I20250809 07:52:57.687556 1439 raft_consensus.cc:1273] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Refusing update from remote peer bd2e4982ad0246ee86f189299d990e77: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250809 07:52:57.688586 1676 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250809 07:52:57.693331 1673 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: true } } unsafe_config_change: true }
I20250809 07:52:57.694779 1439 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: true } } unsafe_config_change: true }
W20250809 07:52:57.697144 1241 consensus_peers.cc:489] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 -> Peer f7857c0655ee471d8cadd2de6452a687 (127.0.212.65:41283): Couldn't send request to peer f7857c0655ee471d8cadd2de6452a687. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4dd206347b14c12a5e03bc206abdf47. This is attempt 1: this message will repeat every 5th retry.
I20250809 07:52:57.700042 1555 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f4dd206347b14c12a5e03bc206abdf47 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250809 07:52:57.702356 1570 catalog_manager.cc:5582] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 reported cstate change: config changed from index 3 to 5, NON_VOTER f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) added. New cstate: current_term: 4 leader_uuid: "bd2e4982ad0246ee86f189299d990e77" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250809 07:52:57.717965 1556 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f4dd206347b14c12a5e03bc206abdf47 on TS 23e69886651c47739e37968aac76d176 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 23e69886651c47739e37968aac76d176
I20250809 07:52:58.280709 1688 ts_tablet_manager.cc:927] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Initiating tablet copy from peer bd2e4982ad0246ee86f189299d990e77 (127.0.212.67:42555)
I20250809 07:52:58.282716 1688 tablet_copy_client.cc:323] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: tablet copy: Beginning tablet copy session from remote peer at address 127.0.212.67:42555
I20250809 07:52:58.291644 1325 tablet_copy_service.cc:140] P bd2e4982ad0246ee86f189299d990e77: Received BeginTabletCopySession request for tablet f4dd206347b14c12a5e03bc206abdf47 from peer f7857c0655ee471d8cadd2de6452a687 ({username='slave'} at 127.0.212.65:50241)
I20250809 07:52:58.292076 1325 tablet_copy_service.cc:161] P bd2e4982ad0246ee86f189299d990e77: Beginning new tablet copy session on tablet f4dd206347b14c12a5e03bc206abdf47 from peer f7857c0655ee471d8cadd2de6452a687 at {username='slave'} at 127.0.212.65:50241: session id = f7857c0655ee471d8cadd2de6452a687-f4dd206347b14c12a5e03bc206abdf47
I20250809 07:52:58.296959 1325 tablet_copy_source_session.cc:215] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 07:52:58.301390 1688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dd206347b14c12a5e03bc206abdf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:52:58.318668 1688 tablet_copy_client.cc:806] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: tablet copy: Starting download of 0 data blocks...
I20250809 07:52:58.319186 1688 tablet_copy_client.cc:670] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: tablet copy: Starting download of 1 WAL segments...
I20250809 07:52:58.322669 1688 tablet_copy_client.cc:538] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 07:52:58.327926 1688 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Bootstrap starting.
I20250809 07:52:58.339206 1688 log.cc:826] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Log is configured to *not* fsync() on all Append() calls
I20250809 07:52:58.349342 1688 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:52:58.349944 1688 tablet_bootstrap.cc:492] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Bootstrap complete.
I20250809 07:52:58.350451 1688 ts_tablet_manager.cc:1397] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20250809 07:52:58.366704 1688 raft_consensus.cc:357] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: true } } unsafe_config_change: true
I20250809 07:52:58.367511 1688 raft_consensus.cc:738] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: f7857c0655ee471d8cadd2de6452a687, State: Initialized, Role: LEARNER
I20250809 07:52:58.368125 1688 consensus_queue.cc:260] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: true } } unsafe_config_change: true
I20250809 07:52:58.371408 1688 ts_tablet_manager.cc:1428] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687: Time spent starting tablet: real 0.021s user 0.022s sys 0.001s
I20250809 07:52:58.372948 1325 tablet_copy_service.cc:342] P bd2e4982ad0246ee86f189299d990e77: Request end of tablet copy session f7857c0655ee471d8cadd2de6452a687-f4dd206347b14c12a5e03bc206abdf47 received from {username='slave'} at 127.0.212.65:50241
I20250809 07:52:58.373258 1325 tablet_copy_service.cc:434] P bd2e4982ad0246ee86f189299d990e77: ending tablet copy session f7857c0655ee471d8cadd2de6452a687-f4dd206347b14c12a5e03bc206abdf47 on tablet f4dd206347b14c12a5e03bc206abdf47 with peer f7857c0655ee471d8cadd2de6452a687
I20250809 07:52:58.878433 1039 raft_consensus.cc:1215] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250809 07:52:58.883797 1556 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f4dd206347b14c12a5e03bc206abdf47 on TS 23e69886651c47739e37968aac76d176 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 23e69886651c47739e37968aac76d176
I20250809 07:52:59.439052 1696 raft_consensus.cc:1062] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77: attempting to promote NON_VOTER f7857c0655ee471d8cadd2de6452a687 to VOTER
I20250809 07:52:59.440534 1696 consensus_queue.cc:237] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: false } } unsafe_config_change: true
I20250809 07:52:59.445021 1039 raft_consensus.cc:1273] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687 [term 4 LEARNER]: Refusing update from remote peer bd2e4982ad0246ee86f189299d990e77: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250809 07:52:59.446278 1699 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250809 07:52:59.446642 1439 raft_consensus.cc:1273] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Refusing update from remote peer bd2e4982ad0246ee86f189299d990e77: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250809 07:52:59.448236 1699 consensus_queue.cc:1035] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250809 07:52:59.452997 1696 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: false } } unsafe_config_change: true }
I20250809 07:52:59.454567 1039 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P f7857c0655ee471d8cadd2de6452a687 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: false } } unsafe_config_change: true }
I20250809 07:52:59.456073 1439 raft_consensus.cc:2953] T f4dd206347b14c12a5e03bc206abdf47 P 6b397062c5a646189e8cb40152ecf781 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: false } } unsafe_config_change: true }
I20250809 07:52:59.463603 1570 catalog_manager.cc:5582] T f4dd206347b14c12a5e03bc206abdf47 P bd2e4982ad0246ee86f189299d990e77 reported cstate change: config changed from index 5 to 6, f7857c0655ee471d8cadd2de6452a687 (127.0.212.65) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "bd2e4982ad0246ee86f189299d990e77" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b397062c5a646189e8cb40152ecf781" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 43721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2e4982ad0246ee86f189299d990e77" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7857c0655ee471d8cadd2de6452a687" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 41283 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250809 07:52:59.559666 849 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250809 07:52:59.576332 849 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "f4dd206347b14c12a5e03bc206abdf47"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "6b397062c5a646189e8cb40152ecf781"
rpc_addresses {
host: "127.0.212.68"
port: 43721
}
}
ts_infos {
permanent_uuid: "bd2e4982ad0246ee86f189299d990e77"
rpc_addresses {
host: "127.0.212.67"
port: 42555
}
}
ts_infos {
permanent_uuid: "f7857c0655ee471d8cadd2de6452a687"
rpc_addresses {
host: "127.0.212.65"
port: 41283
}
}
I20250809 07:52:59.578348 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 955
I20250809 07:52:59.597537 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1221
I20250809 07:52:59.618917 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1354
I20250809 07:52:59.638783 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1525
2025-08-09T07:52:59Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19327 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250809 07:52:59.691844 849 test_util.cc:276] Using random seed: -88958164
I20250809 07:52:59.697587 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:52:59.697743 849 ts_itest-base.cc:116] --------------
I20250809 07:52:59.697860 849 ts_itest-base.cc:117] 3 tablet servers
I20250809 07:52:59.697963 849 ts_itest-base.cc:118] 3 replicas per TS
I20250809 07:52:59.698077 849 ts_itest-base.cc:119] --------------
2025-08-09T07:52:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:52:59Z Disabled control of system clock
I20250809 07:52:59.728626 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:45939
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:45939
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250809 07:53:00.010141 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:00.010663 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:00.011067 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:00.041443 1719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:53:00.041723 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:00.041930 1719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:53:00.042124 1719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:53:00.076778 1719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:45939
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:00.077981 1719 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:00.079448 1719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:00.089784 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:00.090463 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:01.182399 1727 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1090 milliseconds
W20250809 07:53:01.183996 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:01.186932 1719 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.097s user 0.001s sys 0.005s
W20250809 07:53:01.187265 1719 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.098s user 0.001s sys 0.005s
I20250809 07:53:01.187569 1719 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:01.188973 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:01.192054 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:01.193552 1719 hybrid_clock.cc:648] HybridClock initialized: now 1754725981193505 us; error 47 us; skew 500 ppm
I20250809 07:53:01.194603 1719 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:01.202746 1719 webserver.cc:489] Webserver started at http://127.0.212.126:38611/ using document root <none> and password file <none>
I20250809 07:53:01.204042 1719 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:01.204341 1719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:01.204901 1719 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:01.211112 1719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b3e042fc0f974d6890a217d6b20b0a6d"
format_stamp: "Formatted at 2025-08-09 07:53:01 on dist-test-slave-tkp6"
I20250809 07:53:01.212568 1719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b3e042fc0f974d6890a217d6b20b0a6d"
format_stamp: "Formatted at 2025-08-09 07:53:01 on dist-test-slave-tkp6"
I20250809 07:53:01.222059 1719 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20250809 07:53:01.229342 1736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:01.230475 1719 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250809 07:53:01.230886 1719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b3e042fc0f974d6890a217d6b20b0a6d"
format_stamp: "Formatted at 2025-08-09 07:53:01 on dist-test-slave-tkp6"
I20250809 07:53:01.231292 1719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:01.301584 1719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:01.303025 1719 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:01.303469 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:01.368153 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:45939
I20250809 07:53:01.368230 1787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:45939 every 8 connection(s)
I20250809 07:53:01.370709 1719 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:53:01.375404 1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:01.377990 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1719
I20250809 07:53:01.378614 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:53:01.396832 1788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d: Bootstrap starting.
I20250809 07:53:01.402707 1788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:01.404402 1788 log.cc:826] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:01.408473 1788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d: No bootstrap required, opened a new log
I20250809 07:53:01.425206 1788 raft_consensus.cc:357] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 45939 } }
I20250809 07:53:01.425804 1788 raft_consensus.cc:383] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:01.426041 1788 raft_consensus.cc:738] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e042fc0f974d6890a217d6b20b0a6d, State: Initialized, Role: FOLLOWER
I20250809 07:53:01.426659 1788 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 45939 } }
I20250809 07:53:01.427127 1788 raft_consensus.cc:397] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:01.427379 1788 raft_consensus.cc:491] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:01.427661 1788 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:01.431445 1788 raft_consensus.cc:513] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 45939 } }
I20250809 07:53:01.432147 1788 leader_election.cc:304] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e042fc0f974d6890a217d6b20b0a6d; no voters:
I20250809 07:53:01.433703 1788 leader_election.cc:290] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:01.434283 1793 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:01.436897 1793 raft_consensus.cc:695] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [term 1 LEADER]: Becoming Leader. State: Replica: b3e042fc0f974d6890a217d6b20b0a6d, State: Running, Role: LEADER
I20250809 07:53:01.437520 1788 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:53:01.437594 1793 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 45939 } }
I20250809 07:53:01.444504 1794 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 45939 } } }
I20250809 07:53:01.445128 1794 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:01.446246 1795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e042fc0f974d6890a217d6b20b0a6d. Latest consensus state: current_term: 1 leader_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e042fc0f974d6890a217d6b20b0a6d" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 45939 } } }
I20250809 07:53:01.446920 1795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:01.449364 1801 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:53:01.461230 1801 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:53:01.474645 1801 catalog_manager.cc:1349] Generated new cluster ID: 252e4806cc2b47debde146643e94f8f7
I20250809 07:53:01.474957 1801 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:53:01.499141 1801 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:53:01.501098 1801 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:53:01.520044 1801 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b3e042fc0f974d6890a217d6b20b0a6d: Generated new TSK 0
I20250809 07:53:01.521067 1801 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:53:01.534024 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:45939
--builtin_ntp_servers=127.0.212.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250809 07:53:01.823815 1812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250809 07:53:01.824437 1812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:01.824685 1812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:01.825168 1812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:01.855898 1812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:01.856724 1812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:53:01.892182 1812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:45939
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:01.893496 1812 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:01.895040 1812 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:01.906692 1818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:01.911937 1821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:01.909185 1819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:03.014710 1820 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1101 milliseconds
I20250809 07:53:03.014804 1812 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:03.016157 1812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:03.020283 1812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:03.021700 1812 hybrid_clock.cc:648] HybridClock initialized: now 1754725983021647 us; error 76 us; skew 500 ppm
I20250809 07:53:03.022540 1812 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:03.029057 1812 webserver.cc:489] Webserver started at http://127.0.212.65:37779/ using document root <none> and password file <none>
I20250809 07:53:03.029968 1812 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:03.030175 1812 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:03.030660 1812 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:03.034991 1812 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "28037a1188a74517853a7db8ef2a48a5"
format_stamp: "Formatted at 2025-08-09 07:53:03 on dist-test-slave-tkp6"
I20250809 07:53:03.036204 1812 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "28037a1188a74517853a7db8ef2a48a5"
format_stamp: "Formatted at 2025-08-09 07:53:03 on dist-test-slave-tkp6"
I20250809 07:53:03.044001 1812 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250809 07:53:03.049889 1828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:03.050968 1812 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250809 07:53:03.051311 1812 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "28037a1188a74517853a7db8ef2a48a5"
format_stamp: "Formatted at 2025-08-09 07:53:03 on dist-test-slave-tkp6"
I20250809 07:53:03.051693 1812 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:03.104401 1812 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:03.105772 1812 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:03.106186 1812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:03.108698 1812 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:03.113344 1812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:03.113592 1812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:03.113888 1812 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:03.114097 1812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:03.267828 1812 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:38825
I20250809 07:53:03.267980 1940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:38825 every 8 connection(s)
I20250809 07:53:03.270402 1812 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:53:03.280959 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1812
I20250809 07:53:03.281426 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:53:03.288193 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:45939
--builtin_ntp_servers=127.0.212.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250809 07:53:03.293519 1941 heartbeater.cc:344] Connected to a master server at 127.0.212.126:45939
I20250809 07:53:03.294023 1941 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:03.295336 1941 heartbeater.cc:507] Master 127.0.212.126:45939 requested a full tablet report, sending...
I20250809 07:53:03.298408 1753 ts_manager.cc:194] Registered new tserver with Master: 28037a1188a74517853a7db8ef2a48a5 (127.0.212.65:38825)
I20250809 07:53:03.301043 1753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:59135
W20250809 07:53:03.591697 1945 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250809 07:53:03.592329 1945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:03.592585 1945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:03.593112 1945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:03.623488 1945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:03.624325 1945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:53:03.657895 1945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:45939
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:03.659121 1945 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:03.660662 1945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250809 07:53:04.305183 1941 heartbeater.cc:499] Master 127.0.212.126:45939 was elected leader, sending a full tablet report...
W20250809 07:53:03.672438 1951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:03.673107 1952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:04.767541 1954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:04.769726 1953 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1092 milliseconds
I20250809 07:53:04.769824 1945 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:04.770932 1945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:04.772990 1945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:04.774348 1945 hybrid_clock.cc:648] HybridClock initialized: now 1754725984774323 us; error 47 us; skew 500 ppm
I20250809 07:53:04.775128 1945 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:04.781534 1945 webserver.cc:489] Webserver started at http://127.0.212.66:44781/ using document root <none> and password file <none>
I20250809 07:53:04.782392 1945 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:04.782600 1945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:04.783025 1945 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:04.787192 1945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d398d05ff78346ffa428d595eb065b3f"
format_stamp: "Formatted at 2025-08-09 07:53:04 on dist-test-slave-tkp6"
I20250809 07:53:04.788237 1945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d398d05ff78346ffa428d595eb065b3f"
format_stamp: "Formatted at 2025-08-09 07:53:04 on dist-test-slave-tkp6"
I20250809 07:53:04.794836 1945 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250809 07:53:04.800240 1961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:04.801182 1945 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250809 07:53:04.801491 1945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d398d05ff78346ffa428d595eb065b3f"
format_stamp: "Formatted at 2025-08-09 07:53:04 on dist-test-slave-tkp6"
I20250809 07:53:04.801820 1945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:04.849318 1945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:04.850703 1945 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:04.851115 1945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:04.853549 1945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:04.857379 1945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:04.857589 1945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:04.857827 1945 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:04.857978 1945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:04.982493 1945 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:38963
I20250809 07:53:04.982601 2073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:38963 every 8 connection(s)
I20250809 07:53:04.985060 1945 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:53:04.994896 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 1945
I20250809 07:53:04.995342 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 07:53:05.001672 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:45939
--builtin_ntp_servers=127.0.212.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250809 07:53:05.010084 2074 heartbeater.cc:344] Connected to a master server at 127.0.212.126:45939
I20250809 07:53:05.010501 2074 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:05.011474 2074 heartbeater.cc:507] Master 127.0.212.126:45939 requested a full tablet report, sending...
I20250809 07:53:05.013602 1753 ts_manager.cc:194] Registered new tserver with Master: d398d05ff78346ffa428d595eb065b3f (127.0.212.66:38963)
I20250809 07:53:05.015213 1753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:34175
W20250809 07:53:05.292488 2078 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250809 07:53:05.293087 2078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:05.293349 2078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:05.293808 2078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:05.325201 2078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:05.326025 2078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:53:05.358968 2078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:45939
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:05.360198 2078 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:05.361668 2078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:05.372730 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:06.018373 2074 heartbeater.cc:499] Master 127.0.212.126:45939 was elected leader, sending a full tablet report...
W20250809 07:53:05.376538 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:05.376724 2078 server_base.cc:1047] running on GCE node
W20250809 07:53:05.377727 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:06.450500 2078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:06.452749 2078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:06.454108 2078 hybrid_clock.cc:648] HybridClock initialized: now 1754725986454065 us; error 55 us; skew 500 ppm
I20250809 07:53:06.454888 2078 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:06.460670 2078 webserver.cc:489] Webserver started at http://127.0.212.67:46703/ using document root <none> and password file <none>
I20250809 07:53:06.461535 2078 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:06.461740 2078 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:06.462183 2078 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:06.466547 2078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "242459a1b18e4c62ae88e4fd6bb1872d"
format_stamp: "Formatted at 2025-08-09 07:53:06 on dist-test-slave-tkp6"
I20250809 07:53:06.467583 2078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "242459a1b18e4c62ae88e4fd6bb1872d"
format_stamp: "Formatted at 2025-08-09 07:53:06 on dist-test-slave-tkp6"
I20250809 07:53:06.474197 2078 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250809 07:53:06.479638 2094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:06.480681 2078 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250809 07:53:06.480986 2078 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "242459a1b18e4c62ae88e4fd6bb1872d"
format_stamp: "Formatted at 2025-08-09 07:53:06 on dist-test-slave-tkp6"
I20250809 07:53:06.481312 2078 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:06.533376 2078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:06.534739 2078 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:06.535137 2078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:06.537549 2078 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:06.541441 2078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:06.541637 2078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:06.541823 2078 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:06.541968 2078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:06.664003 2078 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:35253
I20250809 07:53:06.664079 2206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:35253 every 8 connection(s)
I20250809 07:53:06.666446 2078 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:53:06.670123 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 2078
I20250809 07:53:06.670552 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 07:53:06.686268 2207 heartbeater.cc:344] Connected to a master server at 127.0.212.126:45939
I20250809 07:53:06.686655 2207 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:06.687747 2207 heartbeater.cc:507] Master 127.0.212.126:45939 requested a full tablet report, sending...
I20250809 07:53:06.689863 1753 ts_manager.cc:194] Registered new tserver with Master: 242459a1b18e4c62ae88e4fd6bb1872d (127.0.212.67:35253)
I20250809 07:53:06.691197 1753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:53473
I20250809 07:53:06.704245 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:06.737587 1753 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45312:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250809 07:53:06.755478 1753 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 07:53:06.804522 2009 tablet_service.cc:1468] Processing CreateTablet for tablet 723adecdd248482fb979d921358931ca (DEFAULT_TABLE table=TestTable [id=a15b8fc440a64b2cb976c8530fa4740e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:06.807211 2009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723adecdd248482fb979d921358931ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:06.808317 2142 tablet_service.cc:1468] Processing CreateTablet for tablet 723adecdd248482fb979d921358931ca (DEFAULT_TABLE table=TestTable [id=a15b8fc440a64b2cb976c8530fa4740e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:06.809674 2142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723adecdd248482fb979d921358931ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:06.810271 1876 tablet_service.cc:1468] Processing CreateTablet for tablet 723adecdd248482fb979d921358931ca (DEFAULT_TABLE table=TestTable [id=a15b8fc440a64b2cb976c8530fa4740e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:06.811947 1876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723adecdd248482fb979d921358931ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:06.829602 2226 tablet_bootstrap.cc:492] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: Bootstrap starting.
I20250809 07:53:06.832651 2227 tablet_bootstrap.cc:492] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: Bootstrap starting.
I20250809 07:53:06.836668 2226 tablet_bootstrap.cc:654] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:06.838984 2226 log.cc:826] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:06.839177 2228 tablet_bootstrap.cc:492] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: Bootstrap starting.
I20250809 07:53:06.839994 2227 tablet_bootstrap.cc:654] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:06.842005 2227 log.cc:826] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:06.844609 2226 tablet_bootstrap.cc:492] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: No bootstrap required, opened a new log
I20250809 07:53:06.845003 2226 ts_tablet_manager.cc:1397] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20250809 07:53:06.845474 2228 tablet_bootstrap.cc:654] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:06.847072 2228 log.cc:826] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:06.847158 2227 tablet_bootstrap.cc:492] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: No bootstrap required, opened a new log
I20250809 07:53:06.847615 2227 ts_tablet_manager.cc:1397] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20250809 07:53:06.851924 2228 tablet_bootstrap.cc:492] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: No bootstrap required, opened a new log
I20250809 07:53:06.852394 2228 ts_tablet_manager.cc:1397] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20250809 07:53:06.868731 2228 raft_consensus.cc:357] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.869601 2228 raft_consensus.cc:738] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28037a1188a74517853a7db8ef2a48a5, State: Initialized, Role: FOLLOWER
I20250809 07:53:06.870167 2228 consensus_queue.cc:260] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.869951 2226 raft_consensus.cc:357] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.870755 2226 raft_consensus.cc:738] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242459a1b18e4c62ae88e4fd6bb1872d, State: Initialized, Role: FOLLOWER
I20250809 07:53:06.871418 2226 consensus_queue.cc:260] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.872948 2227 raft_consensus.cc:357] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.873916 2228 ts_tablet_manager.cc:1428] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20250809 07:53:06.873908 2227 raft_consensus.cc:738] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d398d05ff78346ffa428d595eb065b3f, State: Initialized, Role: FOLLOWER
I20250809 07:53:06.874707 2227 consensus_queue.cc:260] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.876176 2207 heartbeater.cc:499] Master 127.0.212.126:45939 was elected leader, sending a full tablet report...
I20250809 07:53:06.878525 2227 ts_tablet_manager.cc:1428] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: Time spent starting tablet: real 0.031s user 0.027s sys 0.002s
I20250809 07:53:06.880089 2226 ts_tablet_manager.cc:1428] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: Time spent starting tablet: real 0.035s user 0.026s sys 0.006s
I20250809 07:53:06.898506 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:06.901563 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 28037a1188a74517853a7db8ef2a48a5 to finish bootstrapping
I20250809 07:53:06.914711 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d398d05ff78346ffa428d595eb065b3f to finish bootstrapping
W20250809 07:53:06.920635 2208 tablet.cc:2378] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:06.924887 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 242459a1b18e4c62ae88e4fd6bb1872d to finish bootstrapping
I20250809 07:53:06.965253 1896 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "723adecdd248482fb979d921358931ca"
dest_uuid: "28037a1188a74517853a7db8ef2a48a5"
from {username='slave'} at 127.0.0.1:41454
I20250809 07:53:06.965762 1896 raft_consensus.cc:491] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250809 07:53:06.966038 1896 raft_consensus.cc:3058] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:06.970224 1896 raft_consensus.cc:513] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:06.972455 1896 leader_election.cc:290] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [CANDIDATE]: Term 1 election: Requested vote from peers d398d05ff78346ffa428d595eb065b3f (127.0.212.66:38963), 242459a1b18e4c62ae88e4fd6bb1872d (127.0.212.67:35253)
I20250809 07:53:06.982668 849 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250809 07:53:06.985191 2029 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723adecdd248482fb979d921358931ca" candidate_uuid: "28037a1188a74517853a7db8ef2a48a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d398d05ff78346ffa428d595eb065b3f"
I20250809 07:53:06.985811 2029 raft_consensus.cc:3058] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:06.985589 2162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723adecdd248482fb979d921358931ca" candidate_uuid: "28037a1188a74517853a7db8ef2a48a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "242459a1b18e4c62ae88e4fd6bb1872d"
I20250809 07:53:06.986263 2162 raft_consensus.cc:3058] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:06.990164 2029 raft_consensus.cc:2466] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28037a1188a74517853a7db8ef2a48a5 in term 1.
W20250809 07:53:06.990747 2075 tablet.cc:2378] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:06.991950 1830 leader_election.cc:304] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28037a1188a74517853a7db8ef2a48a5, d398d05ff78346ffa428d595eb065b3f; no voters:
I20250809 07:53:06.992702 2232 raft_consensus.cc:2802] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:06.992913 2162 raft_consensus.cc:2466] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28037a1188a74517853a7db8ef2a48a5 in term 1.
I20250809 07:53:06.994625 2232 raft_consensus.cc:695] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 LEADER]: Becoming Leader. State: Replica: 28037a1188a74517853a7db8ef2a48a5, State: Running, Role: LEADER
I20250809 07:53:06.995404 2232 consensus_queue.cc:237] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:07.006451 1750 catalog_manager.cc:5582] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28037a1188a74517853a7db8ef2a48a5 (127.0.212.65). New cstate: current_term: 1 leader_uuid: "28037a1188a74517853a7db8ef2a48a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } health_report { overall_health: HEALTHY } } }
W20250809 07:53:07.029999 1942 tablet.cc:2378] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:07.087877 849 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250809 07:53:07.293057 849 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250809 07:53:07.398319 2232 consensus_queue.cc:1035] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:53:07.415452 2239 consensus_queue.cc:1035] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:53:09.124089 1896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "723adecdd248482fb979d921358931ca"
dest_uuid: "28037a1188a74517853a7db8ef2a48a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50502
I20250809 07:53:09.124565 1896 raft_consensus.cc:604] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 LEADER]: Received request to transfer leadership
I20250809 07:53:09.167773 2268 raft_consensus.cc:991] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5: : Instructing follower 242459a1b18e4c62ae88e4fd6bb1872d to start an election
I20250809 07:53:09.168156 2239 raft_consensus.cc:1079] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 LEADER]: Signalling peer 242459a1b18e4c62ae88e4fd6bb1872d to start an election
I20250809 07:53:09.169396 2162 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "723adecdd248482fb979d921358931ca"
dest_uuid: "242459a1b18e4c62ae88e4fd6bb1872d"
from {username='slave'} at 127.0.212.65:52543
I20250809 07:53:09.169843 2162 raft_consensus.cc:491] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250809 07:53:09.170066 2162 raft_consensus.cc:3058] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:53:09.173991 2162 raft_consensus.cc:513] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:09.175948 2162 leader_election.cc:290] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [CANDIDATE]: Term 2 election: Requested vote from peers d398d05ff78346ffa428d595eb065b3f (127.0.212.66:38963), 28037a1188a74517853a7db8ef2a48a5 (127.0.212.65:38825)
I20250809 07:53:09.186401 2029 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723adecdd248482fb979d921358931ca" candidate_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d398d05ff78346ffa428d595eb065b3f"
I20250809 07:53:09.186770 1896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723adecdd248482fb979d921358931ca" candidate_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "28037a1188a74517853a7db8ef2a48a5"
I20250809 07:53:09.186973 2029 raft_consensus.cc:3058] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:53:09.187346 1896 raft_consensus.cc:3053] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 LEADER]: Stepping down as leader of term 1
I20250809 07:53:09.187641 1896 raft_consensus.cc:738] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28037a1188a74517853a7db8ef2a48a5, State: Running, Role: LEADER
I20250809 07:53:09.188287 1896 consensus_queue.cc:260] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:09.189476 1896 raft_consensus.cc:3058] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:53:09.192924 2029 raft_consensus.cc:2466] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242459a1b18e4c62ae88e4fd6bb1872d in term 2.
I20250809 07:53:09.194005 2096 leader_election.cc:304] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242459a1b18e4c62ae88e4fd6bb1872d, d398d05ff78346ffa428d595eb065b3f; no voters:
I20250809 07:53:09.194803 1896 raft_consensus.cc:2466] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242459a1b18e4c62ae88e4fd6bb1872d in term 2.
I20250809 07:53:09.195998 2272 raft_consensus.cc:2802] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:53:09.197153 2272 raft_consensus.cc:695] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [term 2 LEADER]: Becoming Leader. State: Replica: 242459a1b18e4c62ae88e4fd6bb1872d, State: Running, Role: LEADER
I20250809 07:53:09.197824 2272 consensus_queue.cc:237] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } }
I20250809 07:53:09.204569 1753 catalog_manager.cc:5582] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d reported cstate change: term changed from 1 to 2, leader changed from 28037a1188a74517853a7db8ef2a48a5 (127.0.212.65) to 242459a1b18e4c62ae88e4fd6bb1872d (127.0.212.67). New cstate: current_term: 2 leader_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242459a1b18e4c62ae88e4fd6bb1872d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 35253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20250809 07:53:09.778941 2029 raft_consensus.cc:1273] T 723adecdd248482fb979d921358931ca P d398d05ff78346ffa428d595eb065b3f [term 2 FOLLOWER]: Refusing update from remote peer 242459a1b18e4c62ae88e4fd6bb1872d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250809 07:53:09.780068 2279 consensus_queue.cc:1035] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d398d05ff78346ffa428d595eb065b3f" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250809 07:53:09.791417 1896 raft_consensus.cc:1273] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 2 FOLLOWER]: Refusing update from remote peer 242459a1b18e4c62ae88e4fd6bb1872d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250809 07:53:09.793890 2279 consensus_queue.cc:1035] T 723adecdd248482fb979d921358931ca P 242459a1b18e4c62ae88e4fd6bb1872d [LEADER]: Connected to new peer: Peer: permanent_uuid: "28037a1188a74517853a7db8ef2a48a5" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250809 07:53:11.693187 1896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "723adecdd248482fb979d921358931ca"
dest_uuid: "28037a1188a74517853a7db8ef2a48a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50514
I20250809 07:53:11.693625 1896 raft_consensus.cc:604] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 2 FOLLOWER]: Received request to transfer leadership
I20250809 07:53:11.693853 1896 raft_consensus.cc:612] T 723adecdd248482fb979d921358931ca P 28037a1188a74517853a7db8ef2a48a5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250809 07:53:12.728281 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1812
I20250809 07:53:12.748311 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1945
I20250809 07:53:12.769124 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 2078
I20250809 07:53:12.788558 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 1719
2025-08-09T07:53:12Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13145 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250809 07:53:12.837785 849 test_util.cc:276] Using random seed: -75812223
I20250809 07:53:12.841712 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:53:12.841868 849 ts_itest-base.cc:116] --------------
I20250809 07:53:12.841984 849 ts_itest-base.cc:117] 3 tablet servers
I20250809 07:53:12.842085 849 ts_itest-base.cc:118] 3 replicas per TS
I20250809 07:53:12.842185 849 ts_itest-base.cc:119] --------------
2025-08-09T07:53:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:53:12Z Disabled control of system clock
I20250809 07:53:12.873371 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:42353
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:42353 with env {}
W20250809 07:53:13.155009 2321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:13.155531 2321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:13.155947 2321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:13.185685 2321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:53:13.185951 2321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:13.186149 2321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:53:13.186339 2321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:53:13.220027 2321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:42353
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:13.221287 2321 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:13.222852 2321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:13.233124 2327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:13.233846 2328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:14.320595 2329 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250809 07:53:14.321770 2330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:14.321794 2321 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:14.324671 2321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:14.327076 2321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:14.328415 2321 hybrid_clock.cc:648] HybridClock initialized: now 1754725994328379 us; error 35 us; skew 500 ppm
I20250809 07:53:14.329205 2321 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:14.334784 2321 webserver.cc:489] Webserver started at http://127.0.212.126:38199/ using document root <none> and password file <none>
I20250809 07:53:14.335626 2321 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:14.335855 2321 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:14.336270 2321 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:14.340718 2321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cbc4b07d15bd4e3c8569cd97c468c1db"
format_stamp: "Formatted at 2025-08-09 07:53:14 on dist-test-slave-tkp6"
I20250809 07:53:14.341749 2321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cbc4b07d15bd4e3c8569cd97c468c1db"
format_stamp: "Formatted at 2025-08-09 07:53:14 on dist-test-slave-tkp6"
I20250809 07:53:14.348482 2321 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250809 07:53:14.353513 2337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:14.354442 2321 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250809 07:53:14.354741 2321 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cbc4b07d15bd4e3c8569cd97c468c1db"
format_stamp: "Formatted at 2025-08-09 07:53:14 on dist-test-slave-tkp6"
I20250809 07:53:14.355062 2321 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:14.397260 2321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:14.398619 2321 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:14.399044 2321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:14.463471 2321 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:42353
I20250809 07:53:14.463542 2388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:42353 every 8 connection(s)
I20250809 07:53:14.466100 2321 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:53:14.470584 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 2321
I20250809 07:53:14.470929 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:53:14.470919 2389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:14.492888 2389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db: Bootstrap starting.
I20250809 07:53:14.498044 2389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:14.499598 2389 log.cc:826] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:14.503863 2389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db: No bootstrap required, opened a new log
I20250809 07:53:14.520186 2389 raft_consensus.cc:357] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 42353 } }
I20250809 07:53:14.520797 2389 raft_consensus.cc:383] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:14.521046 2389 raft_consensus.cc:738] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc4b07d15bd4e3c8569cd97c468c1db, State: Initialized, Role: FOLLOWER
I20250809 07:53:14.521718 2389 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 42353 } }
I20250809 07:53:14.522217 2389 raft_consensus.cc:397] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:14.522464 2389 raft_consensus.cc:491] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:14.522770 2389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:14.526836 2389 raft_consensus.cc:513] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 42353 } }
I20250809 07:53:14.527467 2389 leader_election.cc:304] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc4b07d15bd4e3c8569cd97c468c1db; no voters:
I20250809 07:53:14.529034 2389 leader_election.cc:290] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:14.529610 2394 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:14.531476 2394 raft_consensus.cc:695] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [term 1 LEADER]: Becoming Leader. State: Replica: cbc4b07d15bd4e3c8569cd97c468c1db, State: Running, Role: LEADER
I20250809 07:53:14.532171 2394 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 42353 } }
I20250809 07:53:14.533135 2389 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:53:14.541043 2396 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc4b07d15bd4e3c8569cd97c468c1db. Latest consensus state: current_term: 1 leader_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 42353 } } }
I20250809 07:53:14.541576 2396 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:14.541536 2395 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc4b07d15bd4e3c8569cd97c468c1db" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 42353 } } }
I20250809 07:53:14.542222 2395 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:14.544293 2402 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:53:14.556229 2402 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:53:14.571254 2402 catalog_manager.cc:1349] Generated new cluster ID: d3f08cf9596440ceb00c701c1d42a47e
I20250809 07:53:14.571476 2402 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:53:14.593286 2402 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:53:14.594609 2402 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:53:14.606884 2402 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cbc4b07d15bd4e3c8569cd97c468c1db: Generated new TSK 0
I20250809 07:53:14.607669 2402 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:53:14.623142 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:42353
--builtin_ntp_servers=127.0.212.84:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 07:53:14.911525 2413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:14.912053 2413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:14.912530 2413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:14.945261 2413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:14.946092 2413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:53:14.980238 2413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:42353
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:14.981524 2413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:14.983067 2413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:14.995355 2419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:14.996395 2420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:16.342926 2422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:16.347272 2413 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.351s user 0.424s sys 0.816s
W20250809 07:53:16.347659 2413 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.352s user 0.424s sys 0.816s
W20250809 07:53:16.352092 2421 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1354 milliseconds
I20250809 07:53:16.352113 2413 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:16.353684 2413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:16.356382 2413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:16.357905 2413 hybrid_clock.cc:648] HybridClock initialized: now 1754725996357808 us; error 78 us; skew 500 ppm
I20250809 07:53:16.358976 2413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:16.367340 2413 webserver.cc:489] Webserver started at http://127.0.212.65:45855/ using document root <none> and password file <none>
I20250809 07:53:16.368587 2413 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:16.368850 2413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:16.369441 2413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:16.376119 2413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1b4a9dd3493d43be91a484219c81f3df"
format_stamp: "Formatted at 2025-08-09 07:53:16 on dist-test-slave-tkp6"
I20250809 07:53:16.377678 2413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1b4a9dd3493d43be91a484219c81f3df"
format_stamp: "Formatted at 2025-08-09 07:53:16 on dist-test-slave-tkp6"
I20250809 07:53:16.387143 2413 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.009s
I20250809 07:53:16.394690 2429 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:16.395993 2413 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20250809 07:53:16.396442 2413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1b4a9dd3493d43be91a484219c81f3df"
format_stamp: "Formatted at 2025-08-09 07:53:16 on dist-test-slave-tkp6"
I20250809 07:53:16.396941 2413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:16.478117 2413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:16.480046 2413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:16.480587 2413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:16.483744 2413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:16.489331 2413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:16.489611 2413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:16.489910 2413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:16.490129 2413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:16.632820 2413 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:35529
I20250809 07:53:16.632876 2541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:35529 every 8 connection(s)
I20250809 07:53:16.636363 2413 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:53:16.637245 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 2413
I20250809 07:53:16.637611 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:53:16.645332 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:42353
--builtin_ntp_servers=127.0.212.84:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:16.677659 2542 heartbeater.cc:344] Connected to a master server at 127.0.212.126:42353
I20250809 07:53:16.678158 2542 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:16.679350 2542 heartbeater.cc:507] Master 127.0.212.126:42353 requested a full tablet report, sending...
I20250809 07:53:16.682242 2354 ts_manager.cc:194] Registered new tserver with Master: 1b4a9dd3493d43be91a484219c81f3df (127.0.212.65:35529)
I20250809 07:53:16.684902 2354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:33895
W20250809 07:53:16.941109 2546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:16.941574 2546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:16.942040 2546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:16.975661 2546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:16.976522 2546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:53:17.010067 2546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:42353
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:17.011346 2546 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:17.012846 2546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:17.024471 2552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:17.688722 2542 heartbeater.cc:499] Master 127.0.212.126:42353 was elected leader, sending a full tablet report...
W20250809 07:53:17.025108 2553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:18.122153 2555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:18.124179 2554 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1096 milliseconds
I20250809 07:53:18.124260 2546 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:18.125391 2546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:18.127432 2546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:18.128754 2546 hybrid_clock.cc:648] HybridClock initialized: now 1754725998128705 us; error 47 us; skew 500 ppm
I20250809 07:53:18.129513 2546 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:18.135147 2546 webserver.cc:489] Webserver started at http://127.0.212.66:39955/ using document root <none> and password file <none>
I20250809 07:53:18.136052 2546 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:18.136256 2546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:18.136698 2546 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:18.140929 2546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7a9dd48f78a443e191ead59b4af8dc0a"
format_stamp: "Formatted at 2025-08-09 07:53:18 on dist-test-slave-tkp6"
I20250809 07:53:18.141959 2546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7a9dd48f78a443e191ead59b4af8dc0a"
format_stamp: "Formatted at 2025-08-09 07:53:18 on dist-test-slave-tkp6"
I20250809 07:53:18.148612 2546 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250809 07:53:18.153756 2562 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:18.154637 2546 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250809 07:53:18.154937 2546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7a9dd48f78a443e191ead59b4af8dc0a"
format_stamp: "Formatted at 2025-08-09 07:53:18 on dist-test-slave-tkp6"
I20250809 07:53:18.155236 2546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:18.203891 2546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:18.205281 2546 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:18.205696 2546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:18.208096 2546 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:18.211910 2546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:18.212116 2546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:18.212343 2546 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:18.212503 2546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:18.333952 2546 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:37961
I20250809 07:53:18.334046 2674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:37961 every 8 connection(s)
I20250809 07:53:18.336488 2546 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:53:18.343616 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 2546
I20250809 07:53:18.344112 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 07:53:18.350615 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:42353
--builtin_ntp_servers=127.0.212.84:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:18.357626 2675 heartbeater.cc:344] Connected to a master server at 127.0.212.126:42353
I20250809 07:53:18.358012 2675 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:18.358949 2675 heartbeater.cc:507] Master 127.0.212.126:42353 requested a full tablet report, sending...
I20250809 07:53:18.360942 2354 ts_manager.cc:194] Registered new tserver with Master: 7a9dd48f78a443e191ead59b4af8dc0a (127.0.212.66:37961)
I20250809 07:53:18.362108 2354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:35913
W20250809 07:53:18.635311 2679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:18.635802 2679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:18.636284 2679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:18.667397 2679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:18.668254 2679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:53:18.702064 2679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:42353
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:18.703322 2679 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:18.704816 2679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:18.716171 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:19.364943 2675 heartbeater.cc:499] Master 127.0.212.126:42353 was elected leader, sending a full tablet report...
W20250809 07:53:18.717389 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:19.810415 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:19.812171 2687 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1092 milliseconds
I20250809 07:53:19.812290 2679 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:19.813390 2679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:19.815802 2679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:19.817201 2679 hybrid_clock.cc:648] HybridClock initialized: now 1754725999817158 us; error 48 us; skew 500 ppm
I20250809 07:53:19.817938 2679 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:19.823872 2679 webserver.cc:489] Webserver started at http://127.0.212.67:39621/ using document root <none> and password file <none>
I20250809 07:53:19.824721 2679 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:19.824918 2679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:19.825356 2679 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:19.829560 2679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b2403fa577214cf79b67f0e62aa28ede"
format_stamp: "Formatted at 2025-08-09 07:53:19 on dist-test-slave-tkp6"
I20250809 07:53:19.830600 2679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b2403fa577214cf79b67f0e62aa28ede"
format_stamp: "Formatted at 2025-08-09 07:53:19 on dist-test-slave-tkp6"
I20250809 07:53:19.837298 2679 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250809 07:53:19.842540 2695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:19.843437 2679 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250809 07:53:19.843760 2679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b2403fa577214cf79b67f0e62aa28ede"
format_stamp: "Formatted at 2025-08-09 07:53:19 on dist-test-slave-tkp6"
I20250809 07:53:19.844058 2679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:19.899320 2679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:19.900715 2679 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:19.901124 2679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:19.903496 2679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:19.907315 2679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:19.907508 2679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:19.907773 2679 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:19.907922 2679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:20.030593 2679 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:39781
I20250809 07:53:20.030687 2807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:39781 every 8 connection(s)
I20250809 07:53:20.033028 2679 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:53:20.040812 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 2679
I20250809 07:53:20.041157 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 07:53:20.052913 2808 heartbeater.cc:344] Connected to a master server at 127.0.212.126:42353
I20250809 07:53:20.053306 2808 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:20.054239 2808 heartbeater.cc:507] Master 127.0.212.126:42353 requested a full tablet report, sending...
I20250809 07:53:20.056237 2354 ts_manager.cc:194] Registered new tserver with Master: b2403fa577214cf79b67f0e62aa28ede (127.0.212.67:39781)
I20250809 07:53:20.057649 2354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:55701
I20250809 07:53:20.060588 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:20.087378 2354 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36104:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250809 07:53:20.105074 2354 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 07:53:20.155812 2610 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae7406df7d140f79deaa000ef8e2c0e (DEFAULT_TABLE table=TestTable [id=1d821735074242ec959c2724be6b6c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:20.155982 2743 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae7406df7d140f79deaa000ef8e2c0e (DEFAULT_TABLE table=TestTable [id=1d821735074242ec959c2724be6b6c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:20.157650 2610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae7406df7d140f79deaa000ef8e2c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:20.157649 2743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae7406df7d140f79deaa000ef8e2c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:20.157447 2477 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae7406df7d140f79deaa000ef8e2c0e (DEFAULT_TABLE table=TestTable [id=1d821735074242ec959c2724be6b6c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:20.158830 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae7406df7d140f79deaa000ef8e2c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:20.178920 2827 tablet_bootstrap.cc:492] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: Bootstrap starting.
I20250809 07:53:20.182773 2828 tablet_bootstrap.cc:492] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: Bootstrap starting.
I20250809 07:53:20.185259 2829 tablet_bootstrap.cc:492] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: Bootstrap starting.
I20250809 07:53:20.187393 2827 tablet_bootstrap.cc:654] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:20.189692 2827 log.cc:826] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:20.189839 2828 tablet_bootstrap.cc:654] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:20.190850 2829 tablet_bootstrap.cc:654] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:20.192147 2828 log.cc:826] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:20.192989 2829 log.cc:826] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:20.195147 2827 tablet_bootstrap.cc:492] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: No bootstrap required, opened a new log
I20250809 07:53:20.195657 2827 ts_tablet_manager.cc:1397] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20250809 07:53:20.197225 2828 tablet_bootstrap.cc:492] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: No bootstrap required, opened a new log
I20250809 07:53:20.197705 2828 ts_tablet_manager.cc:1397] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20250809 07:53:20.198128 2829 tablet_bootstrap.cc:492] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: No bootstrap required, opened a new log
I20250809 07:53:20.198496 2829 ts_tablet_manager.cc:1397] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250809 07:53:20.214803 2829 raft_consensus.cc:357] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.215448 2829 raft_consensus.cc:383] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:20.215701 2829 raft_consensus.cc:738] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2403fa577214cf79b67f0e62aa28ede, State: Initialized, Role: FOLLOWER
I20250809 07:53:20.216452 2829 consensus_queue.cc:260] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.220196 2808 heartbeater.cc:499] Master 127.0.212.126:42353 was elected leader, sending a full tablet report...
I20250809 07:53:20.221731 2827 raft_consensus.cc:357] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.222633 2827 raft_consensus.cc:383] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:20.222434 2828 raft_consensus.cc:357] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.223053 2829 ts_tablet_manager.cc:1428] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: Time spent starting tablet: real 0.024s user 0.012s sys 0.012s
I20250809 07:53:20.222944 2827 raft_consensus.cc:738] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a9dd3493d43be91a484219c81f3df, State: Initialized, Role: FOLLOWER
I20250809 07:53:20.223290 2828 raft_consensus.cc:383] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:20.223598 2828 raft_consensus.cc:738] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9dd48f78a443e191ead59b4af8dc0a, State: Initialized, Role: FOLLOWER
I20250809 07:53:20.223987 2827 consensus_queue.cc:260] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.224458 2828 consensus_queue.cc:260] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.229104 2828 ts_tablet_manager.cc:1428] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: Time spent starting tablet: real 0.031s user 0.025s sys 0.007s
I20250809 07:53:20.230839 2827 ts_tablet_manager.cc:1428] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: Time spent starting tablet: real 0.035s user 0.026s sys 0.004s
W20250809 07:53:20.287896 2809 tablet.cc:2378] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:20.295887 2835 raft_consensus.cc:491] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:53:20.296337 2835 raft_consensus.cc:513] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.298681 2835 leader_election.cc:290] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4a9dd3493d43be91a484219c81f3df (127.0.212.65:35529), b2403fa577214cf79b67f0e62aa28ede (127.0.212.67:39781)
I20250809 07:53:20.310918 2497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ae7406df7d140f79deaa000ef8e2c0e" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4a9dd3493d43be91a484219c81f3df" is_pre_election: true
I20250809 07:53:20.310962 2763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ae7406df7d140f79deaa000ef8e2c0e" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2403fa577214cf79b67f0e62aa28ede" is_pre_election: true
I20250809 07:53:20.311816 2763 raft_consensus.cc:2466] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 0.
I20250809 07:53:20.311815 2497 raft_consensus.cc:2466] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 0.
I20250809 07:53:20.313119 2563 leader_election.cc:304] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b4a9dd3493d43be91a484219c81f3df, 7a9dd48f78a443e191ead59b4af8dc0a, b2403fa577214cf79b67f0e62aa28ede; no voters:
I20250809 07:53:20.313789 2835 raft_consensus.cc:2802] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:53:20.314038 2835 raft_consensus.cc:491] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:53:20.314276 2835 raft_consensus.cc:3058] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:20.318297 2835 raft_consensus.cc:513] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.319646 2835 leader_election.cc:290] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 election: Requested vote from peers 1b4a9dd3493d43be91a484219c81f3df (127.0.212.65:35529), b2403fa577214cf79b67f0e62aa28ede (127.0.212.67:39781)
I20250809 07:53:20.320333 2497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ae7406df7d140f79deaa000ef8e2c0e" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4a9dd3493d43be91a484219c81f3df"
I20250809 07:53:20.320442 2763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ae7406df7d140f79deaa000ef8e2c0e" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2403fa577214cf79b67f0e62aa28ede"
I20250809 07:53:20.320704 2497 raft_consensus.cc:3058] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:20.320837 2763 raft_consensus.cc:3058] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:20.324934 2763 raft_consensus.cc:2466] T 5ae7406df7d140f79deaa000ef8e2c0e P b2403fa577214cf79b67f0e62aa28ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 1.
I20250809 07:53:20.325178 2497 raft_consensus.cc:2466] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 1.
I20250809 07:53:20.325836 2566 leader_election.cc:304] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9dd48f78a443e191ead59b4af8dc0a, b2403fa577214cf79b67f0e62aa28ede; no voters:
I20250809 07:53:20.326450 2835 raft_consensus.cc:2802] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:20.327891 2835 raft_consensus.cc:695] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [term 1 LEADER]: Becoming Leader. State: Replica: 7a9dd48f78a443e191ead59b4af8dc0a, State: Running, Role: LEADER
I20250809 07:53:20.328608 2835 consensus_queue.cc:237] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.340113 2353 catalog_manager.cc:5582] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9dd48f78a443e191ead59b4af8dc0a (127.0.212.66). New cstate: current_term: 1 leader_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } health_report { overall_health: UNKNOWN } } }
W20250809 07:53:20.342500 2676 tablet.cc:2378] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:20.385957 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:20.388968 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1b4a9dd3493d43be91a484219c81f3df to finish bootstrapping
I20250809 07:53:20.400197 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7a9dd48f78a443e191ead59b4af8dc0a to finish bootstrapping
W20250809 07:53:20.408974 2543 tablet.cc:2378] T 5ae7406df7d140f79deaa000ef8e2c0e P 1b4a9dd3493d43be91a484219c81f3df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:20.409595 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b2403fa577214cf79b67f0e62aa28ede to finish bootstrapping
I20250809 07:53:20.421057 2353 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36104:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250809 07:53:20.422490 2353 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 07:53:20.437067 2477 tablet_service.cc:1468] Processing CreateTablet for tablet 7e39a3d6fc744f7bbd9b9ab975172625 (DEFAULT_TABLE table=TestAnotherTable [id=1deb9347f2cb42e1adfce06ebacc61ca]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250809 07:53:20.438127 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e39a3d6fc744f7bbd9b9ab975172625. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:20.438074 2610 tablet_service.cc:1468] Processing CreateTablet for tablet 7e39a3d6fc744f7bbd9b9ab975172625 (DEFAULT_TABLE table=TestAnotherTable [id=1deb9347f2cb42e1adfce06ebacc61ca]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250809 07:53:20.438520 2743 tablet_service.cc:1468] Processing CreateTablet for tablet 7e39a3d6fc744f7bbd9b9ab975172625 (DEFAULT_TABLE table=TestAnotherTable [id=1deb9347f2cb42e1adfce06ebacc61ca]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250809 07:53:20.439078 2610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e39a3d6fc744f7bbd9b9ab975172625. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:20.439531 2743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e39a3d6fc744f7bbd9b9ab975172625. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:20.452610 2829 tablet_bootstrap.cc:492] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede: Bootstrap starting.
I20250809 07:53:20.452647 2827 tablet_bootstrap.cc:492] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df: Bootstrap starting.
I20250809 07:53:20.454563 2828 tablet_bootstrap.cc:492] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a: Bootstrap starting.
I20250809 07:53:20.457392 2829 tablet_bootstrap.cc:654] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:20.458781 2827 tablet_bootstrap.cc:654] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:20.460551 2828 tablet_bootstrap.cc:654] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:20.463114 2829 tablet_bootstrap.cc:492] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede: No bootstrap required, opened a new log
I20250809 07:53:20.463522 2829 ts_tablet_manager.cc:1397] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250809 07:53:20.465963 2829 raft_consensus.cc:357] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.466673 2829 raft_consensus.cc:383] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:20.466974 2829 raft_consensus.cc:738] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2403fa577214cf79b67f0e62aa28ede, State: Initialized, Role: FOLLOWER
I20250809 07:53:20.469066 2829 consensus_queue.cc:260] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.471352 2829 ts_tablet_manager.cc:1428] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250809 07:53:20.472461 2827 tablet_bootstrap.cc:492] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df: No bootstrap required, opened a new log
I20250809 07:53:20.473045 2827 ts_tablet_manager.cc:1397] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20250809 07:53:20.478930 2828 tablet_bootstrap.cc:492] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a: No bootstrap required, opened a new log
I20250809 07:53:20.479357 2828 ts_tablet_manager.cc:1397] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20250809 07:53:20.479082 2827 raft_consensus.cc:357] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.480401 2827 raft_consensus.cc:383] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:20.480721 2827 raft_consensus.cc:738] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a9dd3493d43be91a484219c81f3df, State: Initialized, Role: FOLLOWER
I20250809 07:53:20.481359 2827 consensus_queue.cc:260] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.483397 2827 ts_tablet_manager.cc:1428] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250809 07:53:20.481969 2828 raft_consensus.cc:357] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.484310 2828 raft_consensus.cc:383] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:20.484634 2828 raft_consensus.cc:738] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9dd48f78a443e191ead59b4af8dc0a, State: Initialized, Role: FOLLOWER
I20250809 07:53:20.485383 2828 consensus_queue.cc:260] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.487401 2828 ts_tablet_manager.cc:1428] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250809 07:53:20.510618 2835 raft_consensus.cc:491] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:53:20.511065 2835 raft_consensus.cc:513] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.512782 2835 leader_election.cc:290] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4a9dd3493d43be91a484219c81f3df (127.0.212.65:35529), b2403fa577214cf79b67f0e62aa28ede (127.0.212.67:39781)
I20250809 07:53:20.513401 2497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e39a3d6fc744f7bbd9b9ab975172625" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4a9dd3493d43be91a484219c81f3df" is_pre_election: true
I20250809 07:53:20.513880 2497 raft_consensus.cc:2466] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 0.
I20250809 07:53:20.513751 2763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e39a3d6fc744f7bbd9b9ab975172625" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2403fa577214cf79b67f0e62aa28ede" is_pre_election: true
I20250809 07:53:20.514278 2763 raft_consensus.cc:2466] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 0.
I20250809 07:53:20.514616 2563 leader_election.cc:304] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4a9dd3493d43be91a484219c81f3df, 7a9dd48f78a443e191ead59b4af8dc0a; no voters:
I20250809 07:53:20.515254 2835 raft_consensus.cc:2802] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:53:20.515537 2835 raft_consensus.cc:491] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:53:20.515816 2835 raft_consensus.cc:3058] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:20.519690 2835 raft_consensus.cc:513] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.521108 2835 leader_election.cc:290] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 election: Requested vote from peers 1b4a9dd3493d43be91a484219c81f3df (127.0.212.65:35529), b2403fa577214cf79b67f0e62aa28ede (127.0.212.67:39781)
I20250809 07:53:20.521689 2497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e39a3d6fc744f7bbd9b9ab975172625" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4a9dd3493d43be91a484219c81f3df"
I20250809 07:53:20.521981 2763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e39a3d6fc744f7bbd9b9ab975172625" candidate_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2403fa577214cf79b67f0e62aa28ede"
I20250809 07:53:20.522166 2497 raft_consensus.cc:3058] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:20.522487 2763 raft_consensus.cc:3058] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:20.527993 2497 raft_consensus.cc:2466] T 7e39a3d6fc744f7bbd9b9ab975172625 P 1b4a9dd3493d43be91a484219c81f3df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 1.
I20250809 07:53:20.528112 2763 raft_consensus.cc:2466] T 7e39a3d6fc744f7bbd9b9ab975172625 P b2403fa577214cf79b67f0e62aa28ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9dd48f78a443e191ead59b4af8dc0a in term 1.
I20250809 07:53:20.528867 2563 leader_election.cc:304] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4a9dd3493d43be91a484219c81f3df, 7a9dd48f78a443e191ead59b4af8dc0a; no voters:
I20250809 07:53:20.529408 2835 raft_consensus.cc:2802] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:20.529723 2835 raft_consensus.cc:695] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [term 1 LEADER]: Becoming Leader. State: Replica: 7a9dd48f78a443e191ead59b4af8dc0a, State: Running, Role: LEADER
I20250809 07:53:20.530354 2835 consensus_queue.cc:237] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } }
I20250809 07:53:20.536471 2352 catalog_manager.cc:5582] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9dd48f78a443e191ead59b4af8dc0a (127.0.212.66). New cstate: current_term: 1 leader_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9dd48f78a443e191ead59b4af8dc0a" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 37961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 } health_report { overall_health: UNKNOWN } } }
I20250809 07:53:20.719713 2835 consensus_queue.cc:1035] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:53:20.735888 2838 consensus_queue.cc:1035] T 5ae7406df7d140f79deaa000ef8e2c0e P 7a9dd48f78a443e191ead59b4af8dc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250809 07:53:20.846856 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:20.847458 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:20.878155 2846 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250809 07:53:20.924175 2835 consensus_queue.cc:1035] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a9dd3493d43be91a484219c81f3df" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:53:20.934665 2850 consensus_queue.cc:1035] T 7e39a3d6fc744f7bbd9b9ab975172625 P 7a9dd48f78a443e191ead59b4af8dc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2403fa577214cf79b67f0e62aa28ede" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250809 07:53:22.085857 2846 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.173s user 0.448s sys 0.721s
W20250809 07:53:22.086242 2846 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.173s user 0.448s sys 0.721s
W20250809 07:53:23.439469 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:23.440078 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:23.469992 2868 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 07:53:24.649246 2868 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.143s user 0.569s sys 0.573s
W20250809 07:53:24.649616 2868 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.144s user 0.569s sys 0.573s
W20250809 07:53:26.005113 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:26.005685 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:26.036650 2883 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 07:53:27.210502 2883 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.137s user 0.463s sys 0.670s
W20250809 07:53:27.210793 2883 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.138s user 0.463s sys 0.670s
W20250809 07:53:28.560096 2898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:28.560673 2898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:28.591235 2898 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 07:53:29.745991 2898 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.116s user 0.478s sys 0.636s
W20250809 07:53:29.746271 2898 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.117s user 0.478s sys 0.636s
I20250809 07:53:30.808395 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 2413
I20250809 07:53:30.829391 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 2546
I20250809 07:53:30.851147 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 2679
I20250809 07:53:30.870041 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 2321
2025-08-09T07:53:30Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18086 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250809 07:53:30.924288 849 test_util.cc:276] Using random seed: -57725719
I20250809 07:53:30.928158 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:53:30.928310 849 ts_itest-base.cc:116] --------------
I20250809 07:53:30.928476 849 ts_itest-base.cc:117] 3 tablet servers
I20250809 07:53:30.928640 849 ts_itest-base.cc:118] 3 replicas per TS
I20250809 07:53:30.928769 849 ts_itest-base.cc:119] --------------
2025-08-09T07:53:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:53:30Z Disabled control of system clock
I20250809 07:53:30.959945 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:40287
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:40287
--superuser_acl=no-such-user with env {}
W20250809 07:53:31.246289 2920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:31.246863 2920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:31.247316 2920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:31.278692 2920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:53:31.278997 2920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:31.279206 2920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:53:31.279402 2920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:53:31.314148 2920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:40287
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:40287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:31.315330 2920 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:31.316912 2920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:31.326812 2926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:31.327864 2927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:32.441424 2929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:32.444200 2928 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1115 milliseconds
I20250809 07:53:32.444300 2920 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:32.445600 2920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:32.448709 2920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:32.450186 2920 hybrid_clock.cc:648] HybridClock initialized: now 1754726012450117 us; error 65 us; skew 500 ppm
I20250809 07:53:32.451167 2920 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:32.457866 2920 webserver.cc:489] Webserver started at http://127.0.212.126:39525/ using document root <none> and password file <none>
I20250809 07:53:32.459102 2920 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:32.459396 2920 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:32.460037 2920 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:32.464274 2920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "45413f7fa7114a2f862112ab909af0cd"
format_stamp: "Formatted at 2025-08-09 07:53:32 on dist-test-slave-tkp6"
I20250809 07:53:32.465276 2920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "45413f7fa7114a2f862112ab909af0cd"
format_stamp: "Formatted at 2025-08-09 07:53:32 on dist-test-slave-tkp6"
I20250809 07:53:32.472003 2920 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250809 07:53:32.477288 2936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:32.478252 2920 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250809 07:53:32.478552 2920 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "45413f7fa7114a2f862112ab909af0cd"
format_stamp: "Formatted at 2025-08-09 07:53:32 on dist-test-slave-tkp6"
I20250809 07:53:32.478852 2920 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:32.522248 2920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:32.523615 2920 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:32.524066 2920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:32.588418 2920 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:40287
I20250809 07:53:32.588480 2987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:40287 every 8 connection(s)
I20250809 07:53:32.590927 2920 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:53:32.595695 2988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:32.596475 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 2920
I20250809 07:53:32.596865 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:53:32.620250 2988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd: Bootstrap starting.
I20250809 07:53:32.625598 2988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:32.627249 2988 log.cc:826] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:32.631448 2988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd: No bootstrap required, opened a new log
I20250809 07:53:32.647239 2988 raft_consensus.cc:357] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45413f7fa7114a2f862112ab909af0cd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 40287 } }
I20250809 07:53:32.647899 2988 raft_consensus.cc:383] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:32.648118 2988 raft_consensus.cc:738] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45413f7fa7114a2f862112ab909af0cd, State: Initialized, Role: FOLLOWER
I20250809 07:53:32.648741 2988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45413f7fa7114a2f862112ab909af0cd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 40287 } }
I20250809 07:53:32.649214 2988 raft_consensus.cc:397] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:32.649461 2988 raft_consensus.cc:491] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:32.649753 2988 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:32.653592 2988 raft_consensus.cc:513] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45413f7fa7114a2f862112ab909af0cd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 40287 } }
I20250809 07:53:32.654253 2988 leader_election.cc:304] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45413f7fa7114a2f862112ab909af0cd; no voters:
I20250809 07:53:32.655814 2988 leader_election.cc:290] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:32.656503 2993 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:32.658982 2993 raft_consensus.cc:695] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [term 1 LEADER]: Becoming Leader. State: Replica: 45413f7fa7114a2f862112ab909af0cd, State: Running, Role: LEADER
I20250809 07:53:32.659603 2993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45413f7fa7114a2f862112ab909af0cd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 40287 } }
I20250809 07:53:32.660295 2988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:53:32.668866 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45413f7fa7114a2f862112ab909af0cd. Latest consensus state: current_term: 1 leader_uuid: "45413f7fa7114a2f862112ab909af0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45413f7fa7114a2f862112ab909af0cd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 40287 } } }
I20250809 07:53:32.669183 2994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45413f7fa7114a2f862112ab909af0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45413f7fa7114a2f862112ab909af0cd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 40287 } } }
I20250809 07:53:32.669505 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:32.669778 2994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:32.673362 3003 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:53:32.682667 3003 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:53:32.697981 3003 catalog_manager.cc:1349] Generated new cluster ID: d4e8b0ca6084497cb60c35f58dd77fdd
I20250809 07:53:32.698273 3003 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:53:32.721239 3003 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:53:32.722554 3003 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:53:32.733994 3003 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 45413f7fa7114a2f862112ab909af0cd: Generated new TSK 0
I20250809 07:53:32.734824 3003 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:53:32.745957 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:40287
--builtin_ntp_servers=127.0.212.84:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 07:53:33.048770 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:33.049261 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:33.049747 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:33.080148 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:33.080953 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:53:33.114781 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:40287
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:33.116221 3012 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:33.117965 3012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:33.129709 3018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:34.531847 3017 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3012
W20250809 07:53:34.629562 3017 kernel_stack_watchdog.cc:198] Thread 3012 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 07:53:34.630487 3012 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.499s user 0.490s sys 0.968s
W20250809 07:53:34.630873 3012 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.500s user 0.491s sys 0.968s
W20250809 07:53:33.130883 3019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:34.632679 3021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:34.635867 3020 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1500 milliseconds
I20250809 07:53:34.635928 3012 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:34.637060 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:34.639081 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:34.640450 3012 hybrid_clock.cc:648] HybridClock initialized: now 1754726014640417 us; error 42 us; skew 500 ppm
I20250809 07:53:34.641189 3012 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:34.646764 3012 webserver.cc:489] Webserver started at http://127.0.212.65:44061/ using document root <none> and password file <none>
I20250809 07:53:34.647639 3012 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:34.647866 3012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:34.648283 3012 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:34.652518 3012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "726ed290c7c047fb8003bdc20e84d853"
format_stamp: "Formatted at 2025-08-09 07:53:34 on dist-test-slave-tkp6"
I20250809 07:53:34.653553 3012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "726ed290c7c047fb8003bdc20e84d853"
format_stamp: "Formatted at 2025-08-09 07:53:34 on dist-test-slave-tkp6"
I20250809 07:53:34.660560 3012 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250809 07:53:34.666115 3028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:34.667186 3012 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250809 07:53:34.667483 3012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "726ed290c7c047fb8003bdc20e84d853"
format_stamp: "Formatted at 2025-08-09 07:53:34 on dist-test-slave-tkp6"
I20250809 07:53:34.667809 3012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:34.716534 3012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:34.717938 3012 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:34.718346 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:34.721091 3012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:34.725039 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:34.725247 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:34.725567 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:34.725723 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:34.872717 3012 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:39215
I20250809 07:53:34.872813 3140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:39215 every 8 connection(s)
I20250809 07:53:34.875357 3012 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:53:34.884181 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3012
I20250809 07:53:34.884563 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:53:34.890232 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:40287
--builtin_ntp_servers=127.0.212.84:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:34.899451 3141 heartbeater.cc:344] Connected to a master server at 127.0.212.126:40287
I20250809 07:53:34.899986 3141 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:34.901218 3141 heartbeater.cc:507] Master 127.0.212.126:40287 requested a full tablet report, sending...
I20250809 07:53:34.904134 2953 ts_manager.cc:194] Registered new tserver with Master: 726ed290c7c047fb8003bdc20e84d853 (127.0.212.65:39215)
I20250809 07:53:34.906723 2953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:57191
W20250809 07:53:35.179951 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:35.180459 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:35.180949 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:35.211351 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:35.212186 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:53:35.246393 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:40287
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:35.247653 3145 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:35.249251 3145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:35.260905 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:35.910678 3141 heartbeater.cc:499] Master 127.0.212.126:40287 was elected leader, sending a full tablet report...
W20250809 07:53:35.261411 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:36.330731 3153 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250809 07:53:36.332010 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:36.332034 3145 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:36.334829 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:36.336913 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:36.338240 3145 hybrid_clock.cc:648] HybridClock initialized: now 1754726016338200 us; error 43 us; skew 500 ppm
I20250809 07:53:36.338991 3145 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:36.344559 3145 webserver.cc:489] Webserver started at http://127.0.212.66:41493/ using document root <none> and password file <none>
I20250809 07:53:36.345413 3145 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:36.345621 3145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:36.346055 3145 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:36.350175 3145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ac96a5b16e8e4b52b680232b18d4df52"
format_stamp: "Formatted at 2025-08-09 07:53:36 on dist-test-slave-tkp6"
I20250809 07:53:36.351214 3145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ac96a5b16e8e4b52b680232b18d4df52"
format_stamp: "Formatted at 2025-08-09 07:53:36 on dist-test-slave-tkp6"
I20250809 07:53:36.357831 3145 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250809 07:53:36.363165 3161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:36.364176 3145 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250809 07:53:36.364485 3145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ac96a5b16e8e4b52b680232b18d4df52"
format_stamp: "Formatted at 2025-08-09 07:53:36 on dist-test-slave-tkp6"
I20250809 07:53:36.364807 3145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:36.417784 3145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:36.419191 3145 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:36.419623 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:36.422055 3145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:36.426036 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:36.426235 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:36.426478 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:36.426623 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:36.551491 3145 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:36201
I20250809 07:53:36.551604 3273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:36201 every 8 connection(s)
I20250809 07:53:36.554106 3145 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:53:36.562848 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3145
I20250809 07:53:36.563328 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 07:53:36.569589 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:40287
--builtin_ntp_servers=127.0.212.84:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:36.575140 3274 heartbeater.cc:344] Connected to a master server at 127.0.212.126:40287
I20250809 07:53:36.575592 3274 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:36.576707 3274 heartbeater.cc:507] Master 127.0.212.126:40287 requested a full tablet report, sending...
I20250809 07:53:36.578716 2953 ts_manager.cc:194] Registered new tserver with Master: ac96a5b16e8e4b52b680232b18d4df52 (127.0.212.66:36201)
I20250809 07:53:36.579931 2953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:58343
W20250809 07:53:36.858282 3278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:36.858772 3278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:36.859264 3278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:36.890419 3278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:36.891249 3278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:53:36.925362 3278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:40287
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:36.926649 3278 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:36.928205 3278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:36.939141 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:37.583350 3274 heartbeater.cc:499] Master 127.0.212.126:40287 was elected leader, sending a full tablet report...
W20250809 07:53:36.939652 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:38.050411 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:38.052718 3278 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.113s user 0.368s sys 0.737s
W20250809 07:53:38.053079 3278 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.114s user 0.368s sys 0.738s
W20250809 07:53:38.053218 3286 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1108 milliseconds
I20250809 07:53:38.053346 3278 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:38.055279 3278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:38.057879 3278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:38.059341 3278 hybrid_clock.cc:648] HybridClock initialized: now 1754726018059286 us; error 46 us; skew 500 ppm
I20250809 07:53:38.060401 3278 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:38.068496 3278 webserver.cc:489] Webserver started at http://127.0.212.67:32837/ using document root <none> and password file <none>
I20250809 07:53:38.069802 3278 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:38.070073 3278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:38.070638 3278 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:38.076874 3278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "706628cc4eb4461bb80cff677b3812f8"
format_stamp: "Formatted at 2025-08-09 07:53:38 on dist-test-slave-tkp6"
I20250809 07:53:38.078308 3278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "706628cc4eb4461bb80cff677b3812f8"
format_stamp: "Formatted at 2025-08-09 07:53:38 on dist-test-slave-tkp6"
I20250809 07:53:38.087271 3278 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250809 07:53:38.094442 3294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:38.095655 3278 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250809 07:53:38.096057 3278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "706628cc4eb4461bb80cff677b3812f8"
format_stamp: "Formatted at 2025-08-09 07:53:38 on dist-test-slave-tkp6"
I20250809 07:53:38.096479 3278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:38.178083 3278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:38.179463 3278 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:38.180037 3278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:38.182597 3278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:38.186964 3278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:38.187157 3278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:38.187371 3278 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:38.187507 3278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:38.310148 3278 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:38843
I20250809 07:53:38.310281 3406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:38843 every 8 connection(s)
I20250809 07:53:38.312662 3278 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:53:38.313133 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3278
I20250809 07:53:38.313477 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 07:53:38.336009 3407 heartbeater.cc:344] Connected to a master server at 127.0.212.126:40287
I20250809 07:53:38.336390 3407 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:38.337299 3407 heartbeater.cc:507] Master 127.0.212.126:40287 requested a full tablet report, sending...
I20250809 07:53:38.339102 2953 ts_manager.cc:194] Registered new tserver with Master: 706628cc4eb4461bb80cff677b3812f8 (127.0.212.67:38843)
I20250809 07:53:38.340287 2953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:45687
I20250809 07:53:38.346030 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:38.373660 2953 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33612:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250809 07:53:38.391214 2953 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 07:53:38.437547 3076 tablet_service.cc:1468] Processing CreateTablet for tablet 8abe03b68f6944c0baa85fecce977d6d (DEFAULT_TABLE table=TestTable [id=0470183e51bf4c91ac4a38212914d298]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:38.439517 3076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8abe03b68f6944c0baa85fecce977d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:38.446599 3209 tablet_service.cc:1468] Processing CreateTablet for tablet 8abe03b68f6944c0baa85fecce977d6d (DEFAULT_TABLE table=TestTable [id=0470183e51bf4c91ac4a38212914d298]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:38.448385 3209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8abe03b68f6944c0baa85fecce977d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:38.448578 3342 tablet_service.cc:1468] Processing CreateTablet for tablet 8abe03b68f6944c0baa85fecce977d6d (DEFAULT_TABLE table=TestTable [id=0470183e51bf4c91ac4a38212914d298]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:38.450294 3342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8abe03b68f6944c0baa85fecce977d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:38.461531 3426 tablet_bootstrap.cc:492] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: Bootstrap starting.
I20250809 07:53:38.467197 3426 tablet_bootstrap.cc:654] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:38.469159 3426 log.cc:826] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:38.474787 3428 tablet_bootstrap.cc:492] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: Bootstrap starting.
I20250809 07:53:38.475056 3426 tablet_bootstrap.cc:492] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: No bootstrap required, opened a new log
I20250809 07:53:38.475513 3426 ts_tablet_manager.cc:1397] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20250809 07:53:38.477433 3429 tablet_bootstrap.cc:492] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: Bootstrap starting.
I20250809 07:53:38.482426 3428 tablet_bootstrap.cc:654] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:38.483217 3429 tablet_bootstrap.cc:654] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:38.484678 3428 log.cc:826] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:38.484846 3429 log.cc:826] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:38.489929 3428 tablet_bootstrap.cc:492] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: No bootstrap required, opened a new log
I20250809 07:53:38.490204 3429 tablet_bootstrap.cc:492] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: No bootstrap required, opened a new log
I20250809 07:53:38.490331 3428 ts_tablet_manager.cc:1397] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20250809 07:53:38.490680 3429 ts_tablet_manager.cc:1397] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20250809 07:53:38.501190 3426 raft_consensus.cc:357] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.502110 3426 raft_consensus.cc:383] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:38.502417 3426 raft_consensus.cc:738] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726ed290c7c047fb8003bdc20e84d853, State: Initialized, Role: FOLLOWER
I20250809 07:53:38.503281 3426 consensus_queue.cc:260] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.509025 3428 raft_consensus.cc:357] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.509938 3428 raft_consensus.cc:383] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:38.508978 3426 ts_tablet_manager.cc:1428] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: Time spent starting tablet: real 0.033s user 0.029s sys 0.000s
I20250809 07:53:38.510285 3428 raft_consensus.cc:738] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac96a5b16e8e4b52b680232b18d4df52, State: Initialized, Role: FOLLOWER
I20250809 07:53:38.511431 3428 consensus_queue.cc:260] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.515472 3428 ts_tablet_manager.cc:1428] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: Time spent starting tablet: real 0.025s user 0.014s sys 0.011s
I20250809 07:53:38.516433 3429 raft_consensus.cc:357] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.517136 3429 raft_consensus.cc:383] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:38.517385 3429 raft_consensus.cc:738] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706628cc4eb4461bb80cff677b3812f8, State: Initialized, Role: FOLLOWER
I20250809 07:53:38.518092 3429 consensus_queue.cc:260] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.520659 3407 heartbeater.cc:499] Master 127.0.212.126:40287 was elected leader, sending a full tablet report...
I20250809 07:53:38.521957 3429 ts_tablet_manager.cc:1428] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: Time spent starting tablet: real 0.031s user 0.024s sys 0.007s
W20250809 07:53:38.560113 3275 tablet.cc:2378] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 07:53:38.567235 3408 tablet.cc:2378] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 07:53:38.633764 3142 tablet.cc:2378] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:53:38.808929 3434 raft_consensus.cc:491] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:53:38.809345 3434 raft_consensus.cc:513] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.811532 3434 leader_election.cc:290] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 726ed290c7c047fb8003bdc20e84d853 (127.0.212.65:39215), ac96a5b16e8e4b52b680232b18d4df52 (127.0.212.66:36201)
I20250809 07:53:38.819587 3229 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8abe03b68f6944c0baa85fecce977d6d" candidate_uuid: "706628cc4eb4461bb80cff677b3812f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac96a5b16e8e4b52b680232b18d4df52" is_pre_election: true
I20250809 07:53:38.820375 3229 raft_consensus.cc:2466] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 706628cc4eb4461bb80cff677b3812f8 in term 0.
I20250809 07:53:38.821458 3296 leader_election.cc:304] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 706628cc4eb4461bb80cff677b3812f8, ac96a5b16e8e4b52b680232b18d4df52; no voters:
I20250809 07:53:38.821786 3096 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8abe03b68f6944c0baa85fecce977d6d" candidate_uuid: "706628cc4eb4461bb80cff677b3812f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "726ed290c7c047fb8003bdc20e84d853" is_pre_election: true
I20250809 07:53:38.822103 3434 raft_consensus.cc:2802] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:53:38.822428 3434 raft_consensus.cc:491] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:53:38.822598 3096 raft_consensus.cc:2466] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 706628cc4eb4461bb80cff677b3812f8 in term 0.
I20250809 07:53:38.822810 3434 raft_consensus.cc:3058] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:38.827299 3434 raft_consensus.cc:513] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.828696 3434 leader_election.cc:290] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [CANDIDATE]: Term 1 election: Requested vote from peers 726ed290c7c047fb8003bdc20e84d853 (127.0.212.65:39215), ac96a5b16e8e4b52b680232b18d4df52 (127.0.212.66:36201)
I20250809 07:53:38.829399 3096 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8abe03b68f6944c0baa85fecce977d6d" candidate_uuid: "706628cc4eb4461bb80cff677b3812f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "726ed290c7c047fb8003bdc20e84d853"
I20250809 07:53:38.829675 3229 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8abe03b68f6944c0baa85fecce977d6d" candidate_uuid: "706628cc4eb4461bb80cff677b3812f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac96a5b16e8e4b52b680232b18d4df52"
I20250809 07:53:38.829888 3096 raft_consensus.cc:3058] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:38.830165 3229 raft_consensus.cc:3058] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:38.836313 3096 raft_consensus.cc:2466] T 8abe03b68f6944c0baa85fecce977d6d P 726ed290c7c047fb8003bdc20e84d853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 706628cc4eb4461bb80cff677b3812f8 in term 1.
I20250809 07:53:38.836313 3229 raft_consensus.cc:2466] T 8abe03b68f6944c0baa85fecce977d6d P ac96a5b16e8e4b52b680232b18d4df52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 706628cc4eb4461bb80cff677b3812f8 in term 1.
I20250809 07:53:38.837307 3297 leader_election.cc:304] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 706628cc4eb4461bb80cff677b3812f8, 726ed290c7c047fb8003bdc20e84d853; no voters:
I20250809 07:53:38.837940 3434 raft_consensus.cc:2802] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:38.839313 3434 raft_consensus.cc:695] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [term 1 LEADER]: Becoming Leader. State: Replica: 706628cc4eb4461bb80cff677b3812f8, State: Running, Role: LEADER
I20250809 07:53:38.840031 3434 consensus_queue.cc:237] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } }
I20250809 07:53:38.848742 2952 catalog_manager.cc:5582] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 706628cc4eb4461bb80cff677b3812f8 (127.0.212.67). New cstate: current_term: 1 leader_uuid: "706628cc4eb4461bb80cff677b3812f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "706628cc4eb4461bb80cff677b3812f8" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 } health_report { overall_health: UNKNOWN } } }
I20250809 07:53:38.899257 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:38.902252 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 726ed290c7c047fb8003bdc20e84d853 to finish bootstrapping
I20250809 07:53:38.913156 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ac96a5b16e8e4b52b680232b18d4df52 to finish bootstrapping
I20250809 07:53:38.922700 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 706628cc4eb4461bb80cff677b3812f8 to finish bootstrapping
I20250809 07:53:39.269982 3434 consensus_queue.cc:1035] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "726ed290c7c047fb8003bdc20e84d853" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 39215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:53:39.286545 3448 consensus_queue.cc:1035] T 8abe03b68f6944c0baa85fecce977d6d P 706628cc4eb4461bb80cff677b3812f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac96a5b16e8e4b52b680232b18d4df52" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 36201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250809 07:53:40.508759 2952 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:48600
W20250809 07:53:41.150249 3403 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac100 after lost signal to thread 3279
W20250809 07:53:41.150974 3403 debug-util.cc:398] Leaking SignalData structure 0x7b080003c880 after lost signal to thread 3406
I20250809 07:53:41.535956 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3012
I20250809 07:53:41.566020 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3145
I20250809 07:53:41.605612 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3278
I20250809 07:53:41.628470 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 2920
2025-08-09T07:53:41Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10755 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250809 07:53:41.679522 849 test_util.cc:276] Using random seed: -46970487
I20250809 07:53:41.683383 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:53:41.683552 849 ts_itest-base.cc:116] --------------
I20250809 07:53:41.683696 849 ts_itest-base.cc:117] 3 tablet servers
I20250809 07:53:41.683864 849 ts_itest-base.cc:118] 3 replicas per TS
I20250809 07:53:41.683997 849 ts_itest-base.cc:119] --------------
2025-08-09T07:53:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:53:41Z Disabled control of system clock
I20250809 07:53:41.714957 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:46717
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:46717 with env {}
W20250809 07:53:42.000308 3477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:42.000821 3477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:42.001267 3477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:42.033035 3477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:53:42.033315 3477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:42.033515 3477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:53:42.033707 3477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:53:42.068367 3477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:46717
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:42.069584 3477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:42.071105 3477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:42.080992 3483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:42.082306 3484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:43.184183 3486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:43.188565 3477 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.107s user 0.353s sys 0.739s
W20250809 07:53:43.188997 3477 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.108s user 0.354s sys 0.739s
W20250809 07:53:43.188989 3485 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1105 milliseconds
I20250809 07:53:43.189357 3477 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:43.190516 3477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:43.193006 3477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:43.194353 3477 hybrid_clock.cc:648] HybridClock initialized: now 1754726023194327 us; error 36 us; skew 500 ppm
I20250809 07:53:43.195132 3477 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:43.201983 3477 webserver.cc:489] Webserver started at http://127.0.212.126:35879/ using document root <none> and password file <none>
I20250809 07:53:43.202839 3477 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:43.203052 3477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:43.203469 3477 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:43.207643 3477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d4947062da8c478fbee8f141047cbfed"
format_stamp: "Formatted at 2025-08-09 07:53:43 on dist-test-slave-tkp6"
I20250809 07:53:43.208732 3477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d4947062da8c478fbee8f141047cbfed"
format_stamp: "Formatted at 2025-08-09 07:53:43 on dist-test-slave-tkp6"
I20250809 07:53:43.216264 3477 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250809 07:53:43.221963 3493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:43.223062 3477 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250809 07:53:43.223383 3477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d4947062da8c478fbee8f141047cbfed"
format_stamp: "Formatted at 2025-08-09 07:53:43 on dist-test-slave-tkp6"
I20250809 07:53:43.223703 3477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:43.293228 3477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:43.294734 3477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:43.295171 3477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:43.360468 3477 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:46717
I20250809 07:53:43.360528 3544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:46717 every 8 connection(s)
I20250809 07:53:43.363051 3477 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:53:43.363879 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3477
I20250809 07:53:43.364364 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:53:43.369300 3545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:43.392676 3545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap starting.
I20250809 07:53:43.397975 3545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:43.399503 3545 log.cc:826] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:43.403508 3545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: No bootstrap required, opened a new log
I20250809 07:53:43.419838 3545 raft_consensus.cc:357] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:43.420480 3545 raft_consensus.cc:383] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:43.420699 3545 raft_consensus.cc:738] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Initialized, Role: FOLLOWER
I20250809 07:53:43.421306 3545 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:43.421794 3545 raft_consensus.cc:397] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:43.422044 3545 raft_consensus.cc:491] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:43.422326 3545 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:43.426368 3545 raft_consensus.cc:513] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:43.426995 3545 leader_election.cc:304] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4947062da8c478fbee8f141047cbfed; no voters:
I20250809 07:53:43.428531 3545 leader_election.cc:290] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:43.429214 3550 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:43.431164 3550 raft_consensus.cc:695] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 1 LEADER]: Becoming Leader. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Running, Role: LEADER
I20250809 07:53:43.431826 3550 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:43.432224 3545 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:53:43.439160 3551 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:53:43.439347 3552 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4947062da8c478fbee8f141047cbfed. Latest consensus state: current_term: 1 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:53:43.439867 3551 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:43.439971 3552 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:43.443231 3558 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:53:43.453397 3558 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:53:43.468293 3558 catalog_manager.cc:1349] Generated new cluster ID: 44b58e310d2a400780112e3bda9e7d64
I20250809 07:53:43.468616 3558 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:53:43.485971 3558 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:53:43.487910 3558 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:53:43.500006 3558 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Generated new TSK 0
I20250809 07:53:43.501027 3558 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:53:43.523866 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:46717
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 07:53:43.812649 3569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:43.813145 3569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:43.813625 3569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:43.844273 3569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:43.845100 3569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:53:43.879060 3569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:43.880445 3569 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:43.882030 3569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:43.895069 3575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:43.900396 3576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:43.904803 3569 server_base.cc:1047] running on GCE node
W20250809 07:53:43.904726 3578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:44.973769 3569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:44.976408 3569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:44.977865 3569 hybrid_clock.cc:648] HybridClock initialized: now 1754726024977823 us; error 51 us; skew 500 ppm
I20250809 07:53:44.978891 3569 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:44.986014 3569 webserver.cc:489] Webserver started at http://127.0.212.65:38675/ using document root <none> and password file <none>
I20250809 07:53:44.987171 3569 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:44.987435 3569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:44.987967 3569 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:44.993367 3569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "928ba1f38b404bedbf94fa16a79809a2"
format_stamp: "Formatted at 2025-08-09 07:53:44 on dist-test-slave-tkp6"
I20250809 07:53:44.994418 3569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "928ba1f38b404bedbf94fa16a79809a2"
format_stamp: "Formatted at 2025-08-09 07:53:44 on dist-test-slave-tkp6"
I20250809 07:53:45.001360 3569 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20250809 07:53:45.006994 3585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:45.008049 3569 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250809 07:53:45.008354 3569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "928ba1f38b404bedbf94fa16a79809a2"
format_stamp: "Formatted at 2025-08-09 07:53:44 on dist-test-slave-tkp6"
I20250809 07:53:45.008689 3569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:45.053229 3569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:45.054600 3569 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:45.055032 3569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:45.057832 3569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:45.062229 3569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:45.062440 3569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:45.062691 3569 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:45.062857 3569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:45.198424 3569 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:37803
I20250809 07:53:45.198508 3697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:37803 every 8 connection(s)
I20250809 07:53:45.200850 3569 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:53:45.206185 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3569
I20250809 07:53:45.206559 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:53:45.212352 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:46717
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:45.221382 3698 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:53:45.221782 3698 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:45.222702 3698 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:53:45.225020 3510 ts_manager.cc:194] Registered new tserver with Master: 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65:37803)
I20250809 07:53:45.226846 3510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:51133
W20250809 07:53:45.493389 3702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:45.493872 3702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:45.494345 3702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:45.524857 3702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:45.525656 3702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:53:45.559077 3702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:45.560328 3702 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:45.561836 3702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:45.572750 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:46.229864 3698 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
W20250809 07:53:46.975440 3707 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3702
W20250809 07:53:47.060770 3707 kernel_stack_watchdog.cc:198] Thread 3702 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 07:53:47.061149 3702 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.487s user 0.585s sys 0.900s
W20250809 07:53:47.061494 3702 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.488s user 0.585s sys 0.900s
W20250809 07:53:45.573720 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:47.061978 3710 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1486 milliseconds
W20250809 07:53:47.063030 3712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:47.063103 3702 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:47.066121 3702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:47.068120 3702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:47.069456 3702 hybrid_clock.cc:648] HybridClock initialized: now 1754726027069415 us; error 47 us; skew 500 ppm
I20250809 07:53:47.070238 3702 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:47.075832 3702 webserver.cc:489] Webserver started at http://127.0.212.66:44735/ using document root <none> and password file <none>
I20250809 07:53:47.076712 3702 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:47.076915 3702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:47.077328 3702 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:47.081574 3702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "55e9bad3aec44a45a7ad11b185083bda"
format_stamp: "Formatted at 2025-08-09 07:53:47 on dist-test-slave-tkp6"
I20250809 07:53:47.082614 3702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "55e9bad3aec44a45a7ad11b185083bda"
format_stamp: "Formatted at 2025-08-09 07:53:47 on dist-test-slave-tkp6"
I20250809 07:53:47.089095 3702 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20250809 07:53:47.094344 3718 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:47.095269 3702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250809 07:53:47.095561 3702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "55e9bad3aec44a45a7ad11b185083bda"
format_stamp: "Formatted at 2025-08-09 07:53:47 on dist-test-slave-tkp6"
I20250809 07:53:47.095902 3702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:47.143339 3702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:47.144778 3702 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:47.145201 3702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:47.147598 3702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:47.151438 3702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:47.151660 3702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:47.151953 3702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:47.152107 3702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:47.278609 3702 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:34191
I20250809 07:53:47.278729 3830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:34191 every 8 connection(s)
I20250809 07:53:47.281127 3702 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:53:47.284757 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3702
I20250809 07:53:47.285243 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 07:53:47.291985 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:46717
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:47.303193 3831 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:53:47.303580 3831 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:47.304559 3831 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:53:47.306586 3510 ts_manager.cc:194] Registered new tserver with Master: 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191)
I20250809 07:53:47.307768 3510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:51003
W20250809 07:53:47.577996 3835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:47.578456 3835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:47.578977 3835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:47.609313 3835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:47.610132 3835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:53:47.643638 3835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:47.644917 3835 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:47.646479 3835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:47.657546 3841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:48.310452 3831 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
W20250809 07:53:47.658102 3842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:48.757609 3844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:48.760140 3843 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1096 milliseconds
W20250809 07:53:48.760248 3835 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.102s user 0.395s sys 0.707s
W20250809 07:53:48.760632 3835 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.103s user 0.395s sys 0.707s
I20250809 07:53:48.760890 3835 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:48.762243 3835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:48.764807 3835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:48.766284 3835 hybrid_clock.cc:648] HybridClock initialized: now 1754726028766239 us; error 29 us; skew 500 ppm
I20250809 07:53:48.767323 3835 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:48.775290 3835 webserver.cc:489] Webserver started at http://127.0.212.67:44787/ using document root <none> and password file <none>
I20250809 07:53:48.776571 3835 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:48.776862 3835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:48.777427 3835 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:53:48.783710 3835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "848d63026bfa48a58b9c402bc70bad0f"
format_stamp: "Formatted at 2025-08-09 07:53:48 on dist-test-slave-tkp6"
I20250809 07:53:48.785156 3835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "848d63026bfa48a58b9c402bc70bad0f"
format_stamp: "Formatted at 2025-08-09 07:53:48 on dist-test-slave-tkp6"
I20250809 07:53:48.794066 3835 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.005s
I20250809 07:53:48.801128 3851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:48.802248 3835 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250809 07:53:48.802630 3835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "848d63026bfa48a58b9c402bc70bad0f"
format_stamp: "Formatted at 2025-08-09 07:53:48 on dist-test-slave-tkp6"
I20250809 07:53:48.803038 3835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:48.866158 3835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:48.867563 3835 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:48.867980 3835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:48.870698 3835 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:48.874485 3835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:53:48.874675 3835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:48.874912 3835 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:53:48.875072 3835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:48.995227 3835 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:39695
I20250809 07:53:48.995316 3963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:39695 every 8 connection(s)
I20250809 07:53:48.997700 3835 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:53:49.007855 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 3835
I20250809 07:53:49.008249 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 07:53:49.022161 3964 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:53:49.022619 3964 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:49.023540 3964 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:53:49.025663 3509 ts_manager.cc:194] Registered new tserver with Master: 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:53:49.027223 3509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:38505
I20250809 07:53:49.027567 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:49.057006 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:53:49.057382 849 test_util.cc:276] Using random seed: -39592620
I20250809 07:53:49.095679 3509 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50912:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250809 07:53:49.133646 3633 tablet_service.cc:1468] Processing CreateTablet for tablet 3e8c3e85a4624afd99ce8b920e6aee14 (DEFAULT_TABLE table=TestTable [id=0b6ccdbc231d4b93a6300c9b7735d38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:49.135041 3633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e8c3e85a4624afd99ce8b920e6aee14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:49.153254 3984 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap starting.
I20250809 07:53:49.158741 3984 tablet_bootstrap.cc:654] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:49.160418 3984 log.cc:826] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:49.164395 3984 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: No bootstrap required, opened a new log
I20250809 07:53:49.164772 3984 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20250809 07:53:49.181177 3984 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:49.181706 3984 raft_consensus.cc:383] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:49.181895 3984 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Initialized, Role: FOLLOWER
I20250809 07:53:49.182466 3984 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:49.182946 3984 raft_consensus.cc:397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:49.183166 3984 raft_consensus.cc:491] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:49.183430 3984 raft_consensus.cc:3058] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:49.187244 3984 raft_consensus.cc:513] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:49.187903 3984 leader_election.cc:304] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 928ba1f38b404bedbf94fa16a79809a2; no voters:
I20250809 07:53:49.189465 3984 leader_election.cc:290] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:49.189862 3986 raft_consensus.cc:2802] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:49.192960 3986 raft_consensus.cc:695] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 LEADER]: Becoming Leader. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Running, Role: LEADER
I20250809 07:53:49.193282 3984 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Time spent starting tablet: real 0.028s user 0.028s sys 0.000s
I20250809 07:53:49.193748 3986 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:49.206008 3509 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65). New cstate: current_term: 1 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } }
I20250809 07:53:49.414281 849 test_util.cc:276] Using random seed: -39235737
I20250809 07:53:49.435822 3503 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50914:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250809 07:53:49.463048 3766 tablet_service.cc:1468] Processing CreateTablet for tablet bdddb55a6cb141aba67217cb66682260 (DEFAULT_TABLE table=TestTable1 [id=080ed2f47af34ef08033603d3bd52be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:49.464460 3766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdddb55a6cb141aba67217cb66682260. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:49.482394 4005 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap starting.
I20250809 07:53:49.487990 4005 tablet_bootstrap.cc:654] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:49.489599 4005 log.cc:826] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:49.493603 4005 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: No bootstrap required, opened a new log
I20250809 07:53:49.493980 4005 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20250809 07:53:49.511765 4005 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:53:49.512524 4005 raft_consensus.cc:383] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:49.512817 4005 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Initialized, Role: FOLLOWER
I20250809 07:53:49.513451 4005 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:53:49.513917 4005 raft_consensus.cc:397] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:49.514153 4005 raft_consensus.cc:491] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:49.514422 4005 raft_consensus.cc:3058] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:49.518692 4005 raft_consensus.cc:513] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:53:49.519356 4005 leader_election.cc:304] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e9bad3aec44a45a7ad11b185083bda; no voters:
I20250809 07:53:49.521193 4005 leader_election.cc:290] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:49.521576 4007 raft_consensus.cc:2802] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:49.523856 4007 raft_consensus.cc:695] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 LEADER]: Becoming Leader. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Running, Role: LEADER
I20250809 07:53:49.524560 4007 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:53:49.524989 4005 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Time spent starting tablet: real 0.031s user 0.023s sys 0.009s
I20250809 07:53:49.534287 3503 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66). New cstate: current_term: 1 leader_uuid: "55e9bad3aec44a45a7ad11b185083bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } health_report { overall_health: HEALTHY } } }
I20250809 07:53:49.714356 849 test_util.cc:276] Using random seed: -38935658
I20250809 07:53:49.733094 3505 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50920:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250809 07:53:49.759909 3899 tablet_service.cc:1468] Processing CreateTablet for tablet 31b5e6b717094538b635418648976f3b (DEFAULT_TABLE table=TestTable2 [id=309d7dc198b946ee949ccbbe82b375fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:53:49.761314 3899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b5e6b717094538b635418648976f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:53:49.779500 4026 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:53:49.784724 4026 tablet_bootstrap.cc:654] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Neither blocks nor log segments found. Creating new log.
I20250809 07:53:49.786312 4026 log.cc:826] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:49.790407 4026 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: No bootstrap required, opened a new log
I20250809 07:53:49.790788 4026 ts_tablet_manager.cc:1397] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20250809 07:53:49.806941 4026 raft_consensus.cc:357] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:53:49.807497 4026 raft_consensus.cc:383] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:53:49.807754 4026 raft_consensus.cc:738] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: FOLLOWER
I20250809 07:53:49.808423 4026 consensus_queue.cc:260] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:53:49.808910 4026 raft_consensus.cc:397] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:49.809180 4026 raft_consensus.cc:491] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:49.809466 4026 raft_consensus.cc:3058] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:53:49.813310 4026 raft_consensus.cc:513] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:53:49.813953 4026 leader_election.cc:304] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 848d63026bfa48a58b9c402bc70bad0f; no voters:
I20250809 07:53:49.815577 4026 leader_election.cc:290] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:53:49.816104 4028 raft_consensus.cc:2802] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:53:49.818579 3964 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
I20250809 07:53:49.819007 4028 raft_consensus.cc:695] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 LEADER]: Becoming Leader. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Running, Role: LEADER
I20250809 07:53:49.819478 4026 ts_tablet_manager.cc:1428] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.028s user 0.024s sys 0.005s
I20250809 07:53:49.819901 4028 consensus_queue.cc:237] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:53:49.828792 3505 catalog_manager.cc:5582] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67). New cstate: current_term: 1 leader_uuid: "848d63026bfa48a58b9c402bc70bad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } health_report { overall_health: HEALTHY } } }
I20250809 07:53:50.014587 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3477
W20250809 07:53:50.223845 3698 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:46717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:46717: connect: Connection refused (error 111)
W20250809 07:53:50.549841 3831 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:46717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:46717: connect: Connection refused (error 111)
W20250809 07:53:50.845969 3964 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:46717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:46717: connect: Connection refused (error 111)
I20250809 07:53:54.754326 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3569
I20250809 07:53:54.772233 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3702
I20250809 07:53:54.793802 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 3835
I20250809 07:53:54.815568 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:46717
--webserver_interface=127.0.212.126
--webserver_port=35879
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:46717 with env {}
W20250809 07:53:55.102078 4106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:55.102670 4106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:55.103077 4106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:55.133082 4106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:53:55.133378 4106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:55.133661 4106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:53:55.133908 4106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:53:55.167672 4106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:46717
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=35879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:55.168939 4106 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:55.170431 4106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:55.180661 4112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:55.181557 4113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:56.287371 4115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:56.289513 4114 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250809 07:53:56.289631 4106 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:53:56.290751 4106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:56.293601 4106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:56.294994 4106 hybrid_clock.cc:648] HybridClock initialized: now 1754726036294950 us; error 49 us; skew 500 ppm
I20250809 07:53:56.295790 4106 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:56.301812 4106 webserver.cc:489] Webserver started at http://127.0.212.126:35879/ using document root <none> and password file <none>
I20250809 07:53:56.302661 4106 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:56.302861 4106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:56.310093 4106 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250809 07:53:56.314230 4122 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:56.315186 4106 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250809 07:53:56.315485 4106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d4947062da8c478fbee8f141047cbfed"
format_stamp: "Formatted at 2025-08-09 07:53:43 on dist-test-slave-tkp6"
I20250809 07:53:56.317335 4106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:56.393741 4106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:56.395151 4106 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:56.395566 4106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:56.460630 4106 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:46717
I20250809 07:53:56.460681 4173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:46717 every 8 connection(s)
I20250809 07:53:56.463377 4106 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:53:56.464871 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 4106
I20250809 07:53:56.466367 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:37803
--local_ip_for_outbound_sockets=127.0.212.65
--tserver_master_addrs=127.0.212.126:46717
--webserver_port=38675
--webserver_interface=127.0.212.65
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:56.473745 4174 sys_catalog.cc:263] Verifying existing consensus state
I20250809 07:53:56.478279 4174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap starting.
I20250809 07:53:56.487924 4174 log.cc:826] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:56.532657 4174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:53:56.533422 4174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap complete.
I20250809 07:53:56.554049 4174 raft_consensus.cc:357] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:56.556146 4174 raft_consensus.cc:738] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Initialized, Role: FOLLOWER
I20250809 07:53:56.556877 4174 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:56.557369 4174 raft_consensus.cc:397] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:56.557621 4174 raft_consensus.cc:491] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:56.557909 4174 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:53:56.563478 4174 raft_consensus.cc:513] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:56.564136 4174 leader_election.cc:304] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4947062da8c478fbee8f141047cbfed; no voters:
I20250809 07:53:56.566007 4174 leader_election.cc:290] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 3 election: Requested vote from peers
I20250809 07:53:56.566399 4178 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Leader election won for term 3
I20250809 07:53:56.569108 4178 raft_consensus.cc:695] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 LEADER]: Becoming Leader. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Running, Role: LEADER
I20250809 07:53:56.569936 4178 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:53:56.570344 4174 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:53:56.579249 4180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4947062da8c478fbee8f141047cbfed. Latest consensus state: current_term: 3 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:53:56.580421 4179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:53:56.581247 4179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:56.582512 4180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:53:56.592584 4186 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:53:56.604801 4186 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=309d7dc198b946ee949ccbbe82b375fc]
I20250809 07:53:56.606566 4186 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c8e7ad19053a4820aee7798f95041711]
I20250809 07:53:56.608219 4186 catalog_manager.cc:671] Loaded metadata for table TestTable [id=e781cfbf096a47f18912579ca163dc0a]
I20250809 07:53:56.615866 4186 tablet_loader.cc:96] loaded metadata for tablet 31b5e6b717094538b635418648976f3b (table TestTable2 [id=309d7dc198b946ee949ccbbe82b375fc])
I20250809 07:53:56.617213 4186 tablet_loader.cc:96] loaded metadata for tablet 3e8c3e85a4624afd99ce8b920e6aee14 (table TestTable [id=e781cfbf096a47f18912579ca163dc0a])
I20250809 07:53:56.618381 4186 tablet_loader.cc:96] loaded metadata for tablet bdddb55a6cb141aba67217cb66682260 (table TestTable1 [id=c8e7ad19053a4820aee7798f95041711])
I20250809 07:53:56.619665 4186 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:53:56.624850 4186 catalog_manager.cc:1261] Loaded cluster ID: 44b58e310d2a400780112e3bda9e7d64
I20250809 07:53:56.625169 4186 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:53:56.633268 4186 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:53:56.639009 4186 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Loaded TSK: 0
I20250809 07:53:56.640568 4195 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250809 07:53:56.640805 4186 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250809 07:53:56.791018 4176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:56.791505 4176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:56.792024 4176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:56.840382 4176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:56.841750 4176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:53:56.894865 4176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:37803
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=38675
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:56.896207 4176 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:56.897781 4176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:56.909597 4201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:57.845520 4170 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 4107
W20250809 07:53:57.846280 4170 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7c0 after lost signal to thread 4173
W20250809 07:53:58.312089 4200 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4176
W20250809 07:53:58.483449 4176 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.573s user 0.570s sys 0.881s
W20250809 07:53:58.487694 4176 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.577s user 0.570s sys 0.882s
W20250809 07:53:58.485827 4204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:58.488067 4176 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250809 07:53:58.487700 4203 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1576 milliseconds
W20250809 07:53:56.910683 4202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:53:58.493016 4176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:53:58.496358 4176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:53:58.497798 4176 hybrid_clock.cc:648] HybridClock initialized: now 1754726038497798 us; error 78 us; skew 500 ppm
I20250809 07:53:58.498571 4176 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:53:58.504845 4176 webserver.cc:489] Webserver started at http://127.0.212.65:38675/ using document root <none> and password file <none>
I20250809 07:53:58.505735 4176 fs_manager.cc:362] Metadata directory not provided
I20250809 07:53:58.505966 4176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:53:58.513592 4176 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250809 07:53:58.518301 4211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:53:58.519368 4176 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250809 07:53:58.519667 4176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "928ba1f38b404bedbf94fa16a79809a2"
format_stamp: "Formatted at 2025-08-09 07:53:44 on dist-test-slave-tkp6"
I20250809 07:53:58.521642 4176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:53:58.574659 4176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:53:58.576120 4176 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:53:58.576539 4176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:53:58.579263 4176 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:53:58.585513 4218 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 07:53:58.592799 4176 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 07:53:58.593020 4176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250809 07:53:58.593299 4176 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 07:53:58.598160 4176 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 07:53:58.598439 4176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250809 07:53:58.598824 4218 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap starting.
I20250809 07:53:58.667366 4218 log.cc:826] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Log is configured to *not* fsync() on all Append() calls
I20250809 07:53:58.789000 4218 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:53:58.790097 4218 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap complete.
I20250809 07:53:58.791716 4218 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Time spent bootstrapping tablet: real 0.193s user 0.151s sys 0.032s
I20250809 07:53:58.814399 4218 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:58.817483 4218 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Initialized, Role: FOLLOWER
I20250809 07:53:58.818686 4218 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:58.819363 4218 raft_consensus.cc:397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:53:58.819742 4218 raft_consensus.cc:491] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:53:58.820196 4218 raft_consensus.cc:3058] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:53:58.828145 4218 raft_consensus.cc:513] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:58.829056 4218 leader_election.cc:304] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 928ba1f38b404bedbf94fa16a79809a2; no voters:
I20250809 07:53:58.836905 4249 raft_consensus.cc:2802] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:53:58.837301 4218 leader_election.cc:290] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 07:53:58.851154 4249 raft_consensus.cc:695] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEADER]: Becoming Leader. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Running, Role: LEADER
I20250809 07:53:58.853123 4249 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:53:58.856822 4218 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Time spent starting tablet: real 0.065s user 0.040s sys 0.009s
I20250809 07:53:58.982880 4176 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:37803
I20250809 07:53:58.983390 4330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:37803 every 8 connection(s)
I20250809 07:53:58.985575 4176 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:53:58.989509 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 4176
I20250809 07:53:58.991271 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:34191
--local_ip_for_outbound_sockets=127.0.212.66
--tserver_master_addrs=127.0.212.126:46717
--webserver_port=44735
--webserver_interface=127.0.212.66
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:53:59.056757 4331 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:53:59.057214 4331 heartbeater.cc:461] Registering TS with master...
I20250809 07:53:59.058321 4331 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:53:59.062906 4139 ts_manager.cc:194] Registered new tserver with Master: 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65:37803)
I20250809 07:53:59.066818 4139 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: term changed from 0 to 2, leader changed from <none> to 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65), VOTER 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) added. New cstate: current_term: 2 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } }
W20250809 07:53:59.148635 4139 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with cas_config_opid_index -1: no extra replica candidate found for tablet 3e8c3e85a4624afd99ce8b920e6aee14 (table TestTable [id=e781cfbf096a47f18912579ca163dc0a]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250809 07:53:59.150977 4139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:35209
I20250809 07:53:59.155238 4331 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
W20250809 07:53:59.462726 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:53:59.463399 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:53:59.464214 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:53:59.517539 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:53:59.518764 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:53:59.560341 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:34191
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=44735
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:53:59.561582 4333 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:53:59.563081 4333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:53:59.578672 4345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:53:59.582698 4346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:00.980368 4344 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4333
W20250809 07:54:01.095176 4333 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.514s user 0.624s sys 0.878s
W20250809 07:54:01.095990 4333 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.514s user 0.625s sys 0.878s
W20250809 07:54:01.096101 4347 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1513 milliseconds
W20250809 07:54:01.097246 4348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:01.097231 4333 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:01.100697 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:01.103154 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:01.104696 4333 hybrid_clock.cc:648] HybridClock initialized: now 1754726041104657 us; error 53 us; skew 500 ppm
I20250809 07:54:01.105420 4333 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:01.110869 4333 webserver.cc:489] Webserver started at http://127.0.212.66:44735/ using document root <none> and password file <none>
I20250809 07:54:01.111692 4333 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:01.111932 4333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:01.119000 4333 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20250809 07:54:01.123294 4356 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:01.124255 4333 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250809 07:54:01.124536 4333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "55e9bad3aec44a45a7ad11b185083bda"
format_stamp: "Formatted at 2025-08-09 07:53:47 on dist-test-slave-tkp6"
I20250809 07:54:01.126346 4333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:01.173074 4333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:01.174448 4333 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:01.174871 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:01.177229 4333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:01.182468 4363 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 07:54:01.189679 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 07:54:01.189954 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250809 07:54:01.190241 4333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 07:54:01.194746 4333 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 07:54:01.194957 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250809 07:54:01.195286 4363 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap starting.
I20250809 07:54:01.246765 4363 log.cc:826] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:01.349057 4333 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:34191
I20250809 07:54:01.349123 4470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:34191 every 8 connection(s)
I20250809 07:54:01.352773 4333 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:54:01.352849 4363 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:01.353761 4363 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap complete.
I20250809 07:54:01.354180 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 4333
I20250809 07:54:01.355393 4363 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Time spent bootstrapping tablet: real 0.160s user 0.121s sys 0.035s
I20250809 07:54:01.356171 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:39695
--local_ip_for_outbound_sockets=127.0.212.67
--tserver_master_addrs=127.0.212.126:46717
--webserver_port=44787
--webserver_interface=127.0.212.67
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:54:01.376427 4363 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:54:01.379854 4471 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:54:01.379791 4363 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Initialized, Role: FOLLOWER
I20250809 07:54:01.380306 4471 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:01.380729 4363 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:54:01.381351 4363 raft_consensus.cc:397] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:54:01.381398 4471 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:54:01.382128 4363 raft_consensus.cc:491] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:54:01.382477 4363 raft_consensus.cc:3058] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:54:01.389835 4363 raft_consensus.cc:513] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:54:01.390676 4363 leader_election.cc:304] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e9bad3aec44a45a7ad11b185083bda; no voters:
I20250809 07:54:01.392735 4363 leader_election.cc:290] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 07:54:01.393149 4476 raft_consensus.cc:2802] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:54:01.394776 4139 ts_manager.cc:194] Registered new tserver with Master: 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191)
I20250809 07:54:01.395846 4363 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Time spent starting tablet: real 0.040s user 0.035s sys 0.005s
I20250809 07:54:01.396943 4476 raft_consensus.cc:695] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEADER]: Becoming Leader. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Running, Role: LEADER
I20250809 07:54:01.397982 4139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:49827
I20250809 07:54:01.397691 4476 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } }
I20250809 07:54:01.401525 4471 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
I20250809 07:54:01.405197 4139 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda reported cstate change: term changed from 0 to 2, leader changed from <none> to 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66), VOTER 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) added. New cstate: current_term: 2 leader_uuid: "55e9bad3aec44a45a7ad11b185083bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } health_report { overall_health: HEALTHY } } }
I20250809 07:54:01.429039 4426 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: true } }
I20250809 07:54:01.432112 4478 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: true } } }
I20250809 07:54:01.440971 4124 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bdddb55a6cb141aba67217cb66682260 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250809 07:54:01.443962 4139 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda reported cstate change: config changed from index -1 to 8, NON_VOTER 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) added. New cstate: current_term: 2 leader_uuid: "55e9bad3aec44a45a7ad11b185083bda" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250809 07:54:01.448144 4360 consensus_peers.cc:489] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda -> Peer 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65:37803): Couldn't send request to peer 928ba1f38b404bedbf94fa16a79809a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdddb55a6cb141aba67217cb66682260. This is attempt 1: this message will repeat every 5th retry.
I20250809 07:54:01.458642 4286 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: NON_VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: true } }
I20250809 07:54:01.464941 4489 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: NON_VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: true } } }
I20250809 07:54:01.473763 4126 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250809 07:54:01.478116 4139 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index -1 to 9, NON_VOTER 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) added. New cstate: current_term: 2 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: NON_VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250809 07:54:01.484429 4213 consensus_peers.cc:489] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 -> Peer 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191): Couldn't send request to peer 55e9bad3aec44a45a7ad11b185083bda. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e8c3e85a4624afd99ce8b920e6aee14. This is attempt 1: this message will repeat every 5th retry.
W20250809 07:54:01.676280 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:01.676776 4473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:01.677251 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:01.707923 4473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:01.708750 4473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:54:01.742559 4473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:39695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=44787
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:01.743850 4473 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:01.745353 4473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:01.756815 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:01.855118 4503 ts_tablet_manager.cc:927] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Initiating tablet copy from peer 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191)
I20250809 07:54:01.857646 4503 tablet_copy_client.cc:323] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.212.66:34191
I20250809 07:54:01.859457 4446 tablet_copy_service.cc:140] P 55e9bad3aec44a45a7ad11b185083bda: Received BeginTabletCopySession request for tablet bdddb55a6cb141aba67217cb66682260 from peer 928ba1f38b404bedbf94fa16a79809a2 ({username='slave'} at 127.0.212.65:43979)
I20250809 07:54:01.859916 4446 tablet_copy_service.cc:161] P 55e9bad3aec44a45a7ad11b185083bda: Beginning new tablet copy session on tablet bdddb55a6cb141aba67217cb66682260 from peer 928ba1f38b404bedbf94fa16a79809a2 at {username='slave'} at 127.0.212.65:43979: session id = 928ba1f38b404bedbf94fa16a79809a2-bdddb55a6cb141aba67217cb66682260
I20250809 07:54:01.864382 4446 tablet_copy_source_session.cc:215] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 07:54:01.867596 4503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdddb55a6cb141aba67217cb66682260. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:01.877306 4503 tablet_copy_client.cc:806] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: tablet copy: Starting download of 0 data blocks...
I20250809 07:54:01.877825 4503 tablet_copy_client.cc:670] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: tablet copy: Starting download of 1 WAL segments...
I20250809 07:54:01.881395 4503 tablet_copy_client.cc:538] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 07:54:01.886435 4503 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap starting.
I20250809 07:54:01.955235 4503 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:01.955837 4503 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap complete.
I20250809 07:54:01.956267 4503 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Time spent bootstrapping tablet: real 0.070s user 0.063s sys 0.008s
I20250809 07:54:01.958397 4503 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: true } }
I20250809 07:54:01.958806 4503 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Initialized, Role: LEARNER
I20250809 07:54:01.959199 4503 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: NON_VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: true } }
I20250809 07:54:01.960498 4503 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250809 07:54:01.962054 4446 tablet_copy_service.cc:342] P 55e9bad3aec44a45a7ad11b185083bda: Request end of tablet copy session 928ba1f38b404bedbf94fa16a79809a2-bdddb55a6cb141aba67217cb66682260 received from {username='slave'} at 127.0.212.65:43979
I20250809 07:54:01.962540 4446 tablet_copy_service.cc:434] P 55e9bad3aec44a45a7ad11b185083bda: ending tablet copy session 928ba1f38b404bedbf94fa16a79809a2-bdddb55a6cb141aba67217cb66682260 on tablet bdddb55a6cb141aba67217cb66682260 with peer 928ba1f38b404bedbf94fa16a79809a2
I20250809 07:54:02.063704 4506 ts_tablet_manager.cc:927] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Initiating tablet copy from peer 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65:37803)
I20250809 07:54:02.065642 4506 tablet_copy_client.cc:323] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: tablet copy: Beginning tablet copy session from remote peer at address 127.0.212.65:37803
I20250809 07:54:02.067380 4306 tablet_copy_service.cc:140] P 928ba1f38b404bedbf94fa16a79809a2: Received BeginTabletCopySession request for tablet 3e8c3e85a4624afd99ce8b920e6aee14 from peer 55e9bad3aec44a45a7ad11b185083bda ({username='slave'} at 127.0.212.66:57167)
I20250809 07:54:02.068054 4306 tablet_copy_service.cc:161] P 928ba1f38b404bedbf94fa16a79809a2: Beginning new tablet copy session on tablet 3e8c3e85a4624afd99ce8b920e6aee14 from peer 55e9bad3aec44a45a7ad11b185083bda at {username='slave'} at 127.0.212.66:57167: session id = 55e9bad3aec44a45a7ad11b185083bda-3e8c3e85a4624afd99ce8b920e6aee14
I20250809 07:54:02.075636 4306 tablet_copy_source_session.cc:215] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 07:54:02.079066 4506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e8c3e85a4624afd99ce8b920e6aee14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:02.090597 4506 tablet_copy_client.cc:806] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: tablet copy: Starting download of 0 data blocks...
I20250809 07:54:02.091109 4506 tablet_copy_client.cc:670] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: tablet copy: Starting download of 1 WAL segments...
I20250809 07:54:02.095430 4506 tablet_copy_client.cc:538] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 07:54:02.100715 4506 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap starting.
I20250809 07:54:02.177949 4506 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:02.178536 4506 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap complete.
I20250809 07:54:02.178942 4506 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Time spent bootstrapping tablet: real 0.078s user 0.067s sys 0.008s
I20250809 07:54:02.180596 4506 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: NON_VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: true } }
I20250809 07:54:02.181082 4506 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Initialized, Role: LEARNER
I20250809 07:54:02.181468 4506 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: NON_VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: true } }
I20250809 07:54:02.182765 4506 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250809 07:54:02.184238 4306 tablet_copy_service.cc:342] P 928ba1f38b404bedbf94fa16a79809a2: Request end of tablet copy session 55e9bad3aec44a45a7ad11b185083bda-3e8c3e85a4624afd99ce8b920e6aee14 received from {username='slave'} at 127.0.212.66:57167
I20250809 07:54:02.184712 4306 tablet_copy_service.cc:434] P 928ba1f38b404bedbf94fa16a79809a2: ending tablet copy session 55e9bad3aec44a45a7ad11b185083bda-3e8c3e85a4624afd99ce8b920e6aee14 on tablet 3e8c3e85a4624afd99ce8b920e6aee14 with peer 55e9bad3aec44a45a7ad11b185083bda
I20250809 07:54:02.261318 4286 raft_consensus.cc:1215] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250809 07:54:02.625873 4426 raft_consensus.cc:1215] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250809 07:54:02.749284 4510 raft_consensus.cc:1062] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: attempting to promote NON_VOTER 928ba1f38b404bedbf94fa16a79809a2 to VOTER
I20250809 07:54:02.750936 4510 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } }
I20250809 07:54:02.755426 4286 raft_consensus.cc:1273] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEARNER]: Refusing update from remote peer 55e9bad3aec44a45a7ad11b185083bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250809 07:54:02.756695 4509 consensus_queue.cc:1035] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250809 07:54:02.762885 4510 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } }
I20250809 07:54:02.764228 4286 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } }
I20250809 07:54:02.773049 4138 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 8 to 9, 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "55e9bad3aec44a45a7ad11b185083bda" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } }
W20250809 07:54:01.757598 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:01.760566 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:02.867852 4499 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 07:54:02.867942 4473 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:02.871641 4473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:02.873772 4473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:02.875108 4473 hybrid_clock.cc:648] HybridClock initialized: now 1754726042875067 us; error 43 us; skew 500 ppm
I20250809 07:54:02.875895 4473 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:02.881587 4473 webserver.cc:489] Webserver started at http://127.0.212.67:44787/ using document root <none> and password file <none>
I20250809 07:54:02.882427 4473 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:02.882644 4473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:02.890357 4473 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250809 07:54:02.894682 4524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:02.895675 4473 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250809 07:54:02.896003 4473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "848d63026bfa48a58b9c402bc70bad0f"
format_stamp: "Formatted at 2025-08-09 07:53:48 on dist-test-slave-tkp6"
I20250809 07:54:02.897910 4473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:02.942770 4473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:02.944185 4473 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:02.944591 4473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:02.946914 4473 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:02.952237 4531 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 07:54:02.959833 4473 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 07:54:02.960043 4473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250809 07:54:02.960304 4473 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 07:54:02.964547 4473 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 07:54:02.964725 4473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.002s
I20250809 07:54:02.965029 4531 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:54:03.025995 4531 log.cc:826] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:03.107929 4531 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:03.108939 4531 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap complete.
I20250809 07:54:03.110611 4531 ts_tablet_manager.cc:1397] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.146s user 0.124s sys 0.018s
I20250809 07:54:03.125687 4531 raft_consensus.cc:357] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:03.128152 4531 raft_consensus.cc:738] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: FOLLOWER
I20250809 07:54:03.129074 4531 consensus_queue.cc:260] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:03.129531 4531 raft_consensus.cc:397] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:54:03.129777 4531 raft_consensus.cc:491] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:54:03.130085 4531 raft_consensus.cc:3058] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:54:03.131142 4473 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:39695
I20250809 07:54:03.131312 4638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:39695 every 8 connection(s)
I20250809 07:54:03.133440 4473 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:54:03.135775 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 4473
I20250809 07:54:03.135876 4531 raft_consensus.cc:513] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:03.136664 4531 leader_election.cc:304] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 848d63026bfa48a58b9c402bc70bad0f; no voters:
I20250809 07:54:03.150355 4531 leader_election.cc:290] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 07:54:03.150758 4643 raft_consensus.cc:2802] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:54:03.155799 4643 raft_consensus.cc:695] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEADER]: Becoming Leader. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Running, Role: LEADER
I20250809 07:54:03.157475 4646 raft_consensus.cc:1062] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: attempting to promote NON_VOTER 55e9bad3aec44a45a7ad11b185083bda to VOTER
I20250809 07:54:03.158310 4643 consensus_queue.cc:237] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:03.159911 4646 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } }
I20250809 07:54:03.163883 4531 ts_tablet_manager.cc:1428] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.053s user 0.027s sys 0.024s
I20250809 07:54:03.166951 4426 raft_consensus.cc:1273] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEARNER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250809 07:54:03.168452 4646 consensus_queue.cc:1035] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250809 07:54:03.173816 4639 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:54:03.174194 4639 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:03.175148 4639 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:54:03.175208 4513 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } }
I20250809 07:54:03.178404 4138 ts_manager.cc:194] Registered new tserver with Master: 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:54:03.178092 4426 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } }
I20250809 07:54:03.179355 4138 catalog_manager.cc:5582] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "848d63026bfa48a58b9c402bc70bad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } health_report { overall_health: HEALTHY } } }
I20250809 07:54:03.186704 4139 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 9 to 10, 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250809 07:54:03.187644 4138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:52787
I20250809 07:54:03.189666 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:54:03.191155 4639 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
I20250809 07:54:03.194878 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:54:03.198529 4286 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } }
W20250809 07:54:03.200603 849 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250809 07:54:03.204779 4426 raft_consensus.cc:1273] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250809 07:54:03.205845 4513 consensus_queue.cc:1035] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250809 07:54:03.211107 4646 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } } }
I20250809 07:54:03.212674 4426 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } } }
W20250809 07:54:03.215783 4214 consensus_peers.cc:489] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 -> Peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Couldn't send request to peer 848d63026bfa48a58b9c402bc70bad0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e8c3e85a4624afd99ce8b920e6aee14. This is attempt 1: this message will repeat every 5th retry.
I20250809 07:54:03.219537 4126 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250809 07:54:03.223368 4137 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 10 to 11, NON_VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added. New cstate: current_term: 2 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250809 07:54:03.424556 4426 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } }
I20250809 07:54:03.429167 4286 raft_consensus.cc:1273] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Refusing update from remote peer 55e9bad3aec44a45a7ad11b185083bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250809 07:54:03.430202 4510 consensus_queue.cc:1035] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250809 07:54:03.435477 4656 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } } }
I20250809 07:54:03.436880 4286 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } } }
W20250809 07:54:03.437429 4359 consensus_peers.cc:489] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda -> Peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Couldn't send request to peer 848d63026bfa48a58b9c402bc70bad0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdddb55a6cb141aba67217cb66682260. This is attempt 1: this message will repeat every 5th retry.
I20250809 07:54:03.442246 4124 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bdddb55a6cb141aba67217cb66682260 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250809 07:54:03.445871 4138 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 9 to 10, NON_VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added. New cstate: current_term: 2 leader_uuid: "55e9bad3aec44a45a7ad11b185083bda" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } } }
I20250809 07:54:03.658583 4125 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bdddb55a6cb141aba67217cb66682260 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250809 07:54:03.693346 4125 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250809 07:54:03.798627 4659 ts_tablet_manager.cc:927] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Initiating tablet copy from peer 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65:37803)
I20250809 07:54:03.800415 4659 tablet_copy_client.cc:323] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.212.65:37803
I20250809 07:54:03.808919 4306 tablet_copy_service.cc:140] P 928ba1f38b404bedbf94fa16a79809a2: Received BeginTabletCopySession request for tablet 3e8c3e85a4624afd99ce8b920e6aee14 from peer 848d63026bfa48a58b9c402bc70bad0f ({username='slave'} at 127.0.212.67:51923)
I20250809 07:54:03.809286 4306 tablet_copy_service.cc:161] P 928ba1f38b404bedbf94fa16a79809a2: Beginning new tablet copy session on tablet 3e8c3e85a4624afd99ce8b920e6aee14 from peer 848d63026bfa48a58b9c402bc70bad0f at {username='slave'} at 127.0.212.67:51923: session id = 848d63026bfa48a58b9c402bc70bad0f-3e8c3e85a4624afd99ce8b920e6aee14
I20250809 07:54:03.813151 4306 tablet_copy_source_session.cc:215] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 07:54:03.815692 4659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e8c3e85a4624afd99ce8b920e6aee14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:03.827558 4659 tablet_copy_client.cc:806] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Starting download of 0 data blocks...
I20250809 07:54:03.828050 4659 tablet_copy_client.cc:670] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Starting download of 1 WAL segments...
I20250809 07:54:03.831275 4659 tablet_copy_client.cc:538] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 07:54:03.836208 4659 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:54:03.913458 4659 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:03.913990 4659 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap complete.
I20250809 07:54:03.914379 4659 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.078s user 0.062s sys 0.014s
I20250809 07:54:03.915961 4659 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } }
I20250809 07:54:03.916432 4659 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: LEARNER
I20250809 07:54:03.916795 4659 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } }
I20250809 07:54:03.919226 4659 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20250809 07:54:03.920738 4306 tablet_copy_service.cc:342] P 928ba1f38b404bedbf94fa16a79809a2: Request end of tablet copy session 848d63026bfa48a58b9c402bc70bad0f-3e8c3e85a4624afd99ce8b920e6aee14 received from {username='slave'} at 127.0.212.67:51923
I20250809 07:54:03.921170 4306 tablet_copy_service.cc:434] P 928ba1f38b404bedbf94fa16a79809a2: ending tablet copy session 848d63026bfa48a58b9c402bc70bad0f-3e8c3e85a4624afd99ce8b920e6aee14 on tablet 3e8c3e85a4624afd99ce8b920e6aee14 with peer 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:03.923154 4659 ts_tablet_manager.cc:927] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Initiating tablet copy from peer 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191)
I20250809 07:54:03.924666 4659 tablet_copy_client.cc:323] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.212.66:34191
I20250809 07:54:03.933861 4446 tablet_copy_service.cc:140] P 55e9bad3aec44a45a7ad11b185083bda: Received BeginTabletCopySession request for tablet bdddb55a6cb141aba67217cb66682260 from peer 848d63026bfa48a58b9c402bc70bad0f ({username='slave'} at 127.0.212.67:49251)
I20250809 07:54:03.934255 4446 tablet_copy_service.cc:161] P 55e9bad3aec44a45a7ad11b185083bda: Beginning new tablet copy session on tablet bdddb55a6cb141aba67217cb66682260 from peer 848d63026bfa48a58b9c402bc70bad0f at {username='slave'} at 127.0.212.67:49251: session id = 848d63026bfa48a58b9c402bc70bad0f-bdddb55a6cb141aba67217cb66682260
I20250809 07:54:03.938266 4446 tablet_copy_source_session.cc:215] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 07:54:03.940279 4659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdddb55a6cb141aba67217cb66682260. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:03.947849 4659 tablet_copy_client.cc:806] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Starting download of 0 data blocks...
I20250809 07:54:03.948262 4659 tablet_copy_client.cc:670] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Starting download of 1 WAL segments...
I20250809 07:54:03.951189 4659 tablet_copy_client.cc:538] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 07:54:03.955289 4659 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:54:04.033859 4659 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:04.034444 4659 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap complete.
I20250809 07:54:04.034862 4659 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.080s user 0.079s sys 0.001s
I20250809 07:54:04.036391 4659 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } }
I20250809 07:54:04.036851 4659 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: LEARNER
I20250809 07:54:04.037220 4659 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: NON_VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: true } }
I20250809 07:54:04.038496 4659 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250809 07:54:04.039958 4446 tablet_copy_service.cc:342] P 55e9bad3aec44a45a7ad11b185083bda: Request end of tablet copy session 848d63026bfa48a58b9c402bc70bad0f-bdddb55a6cb141aba67217cb66682260 received from {username='slave'} at 127.0.212.67:49251
I20250809 07:54:04.040357 4446 tablet_copy_service.cc:434] P 55e9bad3aec44a45a7ad11b185083bda: ending tablet copy session 848d63026bfa48a58b9c402bc70bad0f-bdddb55a6cb141aba67217cb66682260 on tablet bdddb55a6cb141aba67217cb66682260 with peer 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:04.204815 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 928ba1f38b404bedbf94fa16a79809a2 to finish bootstrapping
I20250809 07:54:04.220772 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 55e9bad3aec44a45a7ad11b185083bda to finish bootstrapping
I20250809 07:54:04.230669 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 848d63026bfa48a58b9c402bc70bad0f to finish bootstrapping
I20250809 07:54:04.333945 4594 raft_consensus.cc:1215] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250809 07:54:04.481952 4406 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 07:54:04.485945 4266 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 07:54:04.487123 4594 raft_consensus.cc:1215] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250809 07:54:04.496917 4574 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d4947062da8c478fbee8f141047cbfed | 127.0.212.126:46717 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.212.84:35031 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
55e9bad3aec44a45a7ad11b185083bda | 127.0.212.66:34191 | HEALTHY | <none> | 1 | 0
848d63026bfa48a58b9c402bc70bad0f | 127.0.212.67:39695 | HEALTHY | <none> | 1 | 0
928ba1f38b404bedbf94fa16a79809a2 | 127.0.212.65:37803 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.212.65 | experimental | 127.0.212.65:37803
local_ip_for_outbound_sockets | 127.0.212.66 | experimental | 127.0.212.66:34191
local_ip_for_outbound_sockets | 127.0.212.67 | experimental | 127.0.212.67:39695
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.212.65:37803
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.212.66:34191
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.212.67:39695
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.212.84:35031 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250809 07:54:04.716202 849 log_verifier.cc:126] Checking tablet 31b5e6b717094538b635418648976f3b
I20250809 07:54:04.746238 849 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 31b5e6b717094538b635418648976f3b
I20250809 07:54:04.746477 849 log_verifier.cc:126] Checking tablet 3e8c3e85a4624afd99ce8b920e6aee14
I20250809 07:54:04.768970 4678 raft_consensus.cc:1062] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: attempting to promote NON_VOTER 848d63026bfa48a58b9c402bc70bad0f to VOTER
I20250809 07:54:04.770328 4678 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:04.775168 4426 raft_consensus.cc:1273] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250809 07:54:04.775281 4594 raft_consensus.cc:1273] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250809 07:54:04.776463 4678 consensus_queue.cc:1035] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250809 07:54:04.777293 4679 consensus_queue.cc:1035] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250809 07:54:04.784528 4655 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
I20250809 07:54:04.786656 4426 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
I20250809 07:54:04.791180 4594 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
I20250809 07:54:04.795060 4138 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 11 to 12, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250809 07:54:04.842669 849 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 3e8c3e85a4624afd99ce8b920e6aee14
I20250809 07:54:04.842917 849 log_verifier.cc:126] Checking tablet bdddb55a6cb141aba67217cb66682260
I20250809 07:54:04.925346 849 log_verifier.cc:177] Verified matching terms for 10 ops in tablet bdddb55a6cb141aba67217cb66682260
I20250809 07:54:04.925765 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 4106
I20250809 07:54:04.951781 849 minidump.cc:252] Setting minidump size limit to 20M
I20250809 07:54:04.953667 849 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:04.954627 849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:04.964246 4709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:05.044786 4699 raft_consensus.cc:1062] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: attempting to promote NON_VOTER 848d63026bfa48a58b9c402bc70bad0f to VOTER
I20250809 07:54:05.047004 4699 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:05.071672 4594 raft_consensus.cc:1273] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 LEARNER]: Refusing update from remote peer 55e9bad3aec44a45a7ad11b185083bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250809 07:54:05.073143 4286 raft_consensus.cc:1273] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Refusing update from remote peer 55e9bad3aec44a45a7ad11b185083bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250809 07:54:05.074872 4716 consensus_queue.cc:1035] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250809 07:54:05.075860 4714 consensus_queue.cc:1035] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250809 07:54:05.092428 4714 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
W20250809 07:54:05.108376 4471 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:46717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:46717: connect: Connection refused (error 111)
I20250809 07:54:05.108206 4594 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
I20250809 07:54:05.111167 4285 raft_consensus.cc:2953] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
W20250809 07:54:05.139222 4331 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:46717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:46717: connect: Connection refused (error 111)
W20250809 07:54:05.150895 4639 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:46717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:46717: connect: Connection refused (error 111)
W20250809 07:54:04.964282 4710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:05.152482 4712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:05.152994 849 server_base.cc:1047] running on GCE node
I20250809 07:54:05.154376 849 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250809 07:54:05.154654 849 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250809 07:54:05.154848 849 hybrid_clock.cc:648] HybridClock initialized: now 1754726045154827 us; error 0 us; skew 500 ppm
I20250809 07:54:05.155659 849 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:05.159070 849 webserver.cc:489] Webserver started at http://0.0.0.0:34775/ using document root <none> and password file <none>
I20250809 07:54:05.160094 849 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:05.160334 849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:05.166597 849 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20250809 07:54:05.170799 4726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:05.171761 849 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250809 07:54:05.172108 849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d4947062da8c478fbee8f141047cbfed"
format_stamp: "Formatted at 2025-08-09 07:53:43 on dist-test-slave-tkp6"
I20250809 07:54:05.174321 849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:05.212021 849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:05.213289 849 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:05.213655 849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:05.221415 849 sys_catalog.cc:263] Verifying existing consensus state
W20250809 07:54:05.224575 849 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.0.212.126:46717 exists but no master address supplied!
I20250809 07:54:05.226186 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap starting.
I20250809 07:54:05.263700 849 log.cc:826] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:05.321313 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=13 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:05.321956 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap complete.
I20250809 07:54:05.334156 849 raft_consensus.cc:357] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:05.334661 849 raft_consensus.cc:738] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Initialized, Role: FOLLOWER
I20250809 07:54:05.335273 849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:05.335682 849 raft_consensus.cc:397] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:54:05.335902 849 raft_consensus.cc:491] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:54:05.336194 849 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 3 FOLLOWER]: Advancing to term 4
I20250809 07:54:05.341356 849 raft_consensus.cc:513] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:05.342003 849 leader_election.cc:304] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4947062da8c478fbee8f141047cbfed; no voters:
I20250809 07:54:05.343071 849 leader_election.cc:290] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 4 election: Requested vote from peers
I20250809 07:54:05.343312 4733 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 4 FOLLOWER]: Leader election won for term 4
I20250809 07:54:05.344864 4733 raft_consensus.cc:695] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 4 LEADER]: Becoming Leader. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Running, Role: LEADER
I20250809 07:54:05.345695 4733 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:05.352875 4735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:54:05.353294 4735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:54:05.354259 4734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4947062da8c478fbee8f141047cbfed. Latest consensus state: current_term: 4 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:54:05.354629 4734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:54:05.388865 849 tablet_replica.cc:331] stopping tablet replica
I20250809 07:54:05.389528 849 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 4 LEADER]: Raft consensus shutting down.
I20250809 07:54:05.390000 849 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 4 FOLLOWER]: Raft consensus is shut down!
I20250809 07:54:05.393416 849 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250809 07:54:05.393982 849 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250809 07:54:05.520208 849 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250809 07:54:10.406257 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 4176
I20250809 07:54:10.426717 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 4333
I20250809 07:54:10.447583 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 4473
I20250809 07:54:10.470420 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:46717
--webserver_interface=127.0.212.126
--webserver_port=35879
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:46717 with env {}
W20250809 07:54:10.762027 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:10.762596 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:10.763049 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:10.793263 4808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:54:10.793593 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:10.793841 4808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:54:10.794078 4808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:54:10.828207 4808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:46717
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=35879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:10.829466 4808 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:10.831010 4808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:10.840952 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:10.841919 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:12.014160 4817 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1167 milliseconds
W20250809 07:54:12.015947 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:12.018079 4808 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.177s user 0.000s sys 0.007s
W20250809 07:54:12.018357 4808 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.177s user 0.000s sys 0.007s
I20250809 07:54:12.018560 4808 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:12.019589 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:12.022058 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:12.023407 4808 hybrid_clock.cc:648] HybridClock initialized: now 1754726052023381 us; error 43 us; skew 500 ppm
I20250809 07:54:12.024232 4808 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:12.031533 4808 webserver.cc:489] Webserver started at http://127.0.212.126:35879/ using document root <none> and password file <none>
I20250809 07:54:12.032431 4808 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:12.032649 4808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:12.040362 4808 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250809 07:54:12.045142 4825 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:12.046262 4808 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250809 07:54:12.046571 4808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d4947062da8c478fbee8f141047cbfed"
format_stamp: "Formatted at 2025-08-09 07:53:43 on dist-test-slave-tkp6"
I20250809 07:54:12.048486 4808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:12.118316 4808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:12.119767 4808 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:12.120199 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:12.186128 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:46717
I20250809 07:54:12.186195 4876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:46717 every 8 connection(s)
I20250809 07:54:12.188869 4808 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:54:12.191843 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 4808
I20250809 07:54:12.193951 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:37803
--local_ip_for_outbound_sockets=127.0.212.65
--tserver_master_addrs=127.0.212.126:46717
--webserver_port=38675
--webserver_interface=127.0.212.65
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:54:12.201704 4877 sys_catalog.cc:263] Verifying existing consensus state
I20250809 07:54:12.207041 4877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap starting.
I20250809 07:54:12.217101 4877 log.cc:826] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:12.290055 4877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=15 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:12.290820 4877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Bootstrap complete.
I20250809 07:54:12.308957 4877 raft_consensus.cc:357] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:12.310957 4877 raft_consensus.cc:738] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Initialized, Role: FOLLOWER
I20250809 07:54:12.311777 4877 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:12.312237 4877 raft_consensus.cc:397] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:54:12.312471 4877 raft_consensus.cc:491] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:54:12.312736 4877 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 5 FOLLOWER]: Advancing to term 6
I20250809 07:54:12.317884 4877 raft_consensus.cc:513] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:12.318456 4877 leader_election.cc:304] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4947062da8c478fbee8f141047cbfed; no voters:
I20250809 07:54:12.320303 4877 leader_election.cc:290] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [CANDIDATE]: Term 6 election: Requested vote from peers
I20250809 07:54:12.320669 4881 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 6 FOLLOWER]: Leader election won for term 6
I20250809 07:54:12.323426 4881 raft_consensus.cc:695] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [term 6 LEADER]: Becoming Leader. State: Replica: d4947062da8c478fbee8f141047cbfed, State: Running, Role: LEADER
I20250809 07:54:12.324230 4881 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } }
I20250809 07:54:12.324699 4877 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:54:12.331032 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4947062da8c478fbee8f141047cbfed. Latest consensus state: current_term: 6 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:54:12.332737 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:54:12.332196 4882 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "d4947062da8c478fbee8f141047cbfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4947062da8c478fbee8f141047cbfed" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 46717 } } }
I20250809 07:54:12.333104 4882 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed [sys.catalog]: This master's current role is: LEADER
I20250809 07:54:12.342743 4888 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:54:12.355422 4888 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=309d7dc198b946ee949ccbbe82b375fc]
I20250809 07:54:12.357075 4888 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c8e7ad19053a4820aee7798f95041711]
I20250809 07:54:12.358600 4888 catalog_manager.cc:671] Loaded metadata for table TestTable [id=f5c17147c1ea45e283b874c44437f3dc]
I20250809 07:54:12.366761 4888 tablet_loader.cc:96] loaded metadata for tablet 31b5e6b717094538b635418648976f3b (table TestTable2 [id=309d7dc198b946ee949ccbbe82b375fc])
I20250809 07:54:12.367985 4888 tablet_loader.cc:96] loaded metadata for tablet 3e8c3e85a4624afd99ce8b920e6aee14 (table TestTable [id=f5c17147c1ea45e283b874c44437f3dc])
I20250809 07:54:12.369318 4888 tablet_loader.cc:96] loaded metadata for tablet bdddb55a6cb141aba67217cb66682260 (table TestTable1 [id=c8e7ad19053a4820aee7798f95041711])
I20250809 07:54:12.370606 4888 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:54:12.375842 4888 catalog_manager.cc:1261] Loaded cluster ID: 44b58e310d2a400780112e3bda9e7d64
I20250809 07:54:12.377079 4888 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:54:12.385106 4888 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:54:12.390410 4888 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d4947062da8c478fbee8f141047cbfed: Loaded TSK: 0
I20250809 07:54:12.391983 4888 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250809 07:54:12.512985 4879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:12.513476 4879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:12.513955 4879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:12.544692 4879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:12.545529 4879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:54:12.578771 4879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:37803
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=38675
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:12.580025 4879 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:12.581557 4879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:12.593238 4904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:13.995394 4903 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4879
W20250809 07:54:14.004325 4903 kernel_stack_watchdog.cc:198] Thread 4879 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 07:54:12.595603 4905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:14.005532 4879 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.412s user 0.507s sys 0.863s
W20250809 07:54:14.005939 4879 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.412s user 0.507s sys 0.864s
W20250809 07:54:14.006783 4907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:14.009375 4879 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250809 07:54:14.009370 4906 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1409 milliseconds
I20250809 07:54:14.010625 4879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:14.012708 4879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:14.014076 4879 hybrid_clock.cc:648] HybridClock initialized: now 1754726054014041 us; error 37 us; skew 500 ppm
I20250809 07:54:14.014853 4879 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:14.020653 4879 webserver.cc:489] Webserver started at http://127.0.212.65:38675/ using document root <none> and password file <none>
I20250809 07:54:14.021519 4879 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:14.021728 4879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:14.029680 4879 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250809 07:54:14.034411 4914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:14.035481 4879 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250809 07:54:14.035808 4879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "928ba1f38b404bedbf94fa16a79809a2"
format_stamp: "Formatted at 2025-08-09 07:53:44 on dist-test-slave-tkp6"
I20250809 07:54:14.037640 4879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:14.091696 4879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:14.093118 4879 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:14.093570 4879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:14.096210 4879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:14.102357 4921 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250809 07:54:14.113152 4879 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250809 07:54:14.113364 4879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250809 07:54:14.113627 4879 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250809 07:54:14.118526 4921 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap starting.
I20250809 07:54:14.122920 4879 ts_tablet_manager.cc:610] Registered 2 tablets
I20250809 07:54:14.123180 4879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.007s sys 0.000s
I20250809 07:54:14.183343 4921 log.cc:826] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:14.281280 4921 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:14.282363 4921 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap complete.
I20250809 07:54:14.284014 4921 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Time spent bootstrapping tablet: real 0.166s user 0.134s sys 0.026s
I20250809 07:54:14.305673 4921 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:14.308974 4921 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Initialized, Role: FOLLOWER
I20250809 07:54:14.310009 4921 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:14.314842 4921 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2: Time spent starting tablet: real 0.030s user 0.025s sys 0.004s
I20250809 07:54:14.315634 4921 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap starting.
I20250809 07:54:14.328179 4879 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:37803
I20250809 07:54:14.328433 5029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:37803 every 8 connection(s)
I20250809 07:54:14.330803 4879 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:54:14.340963 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 4879
I20250809 07:54:14.342828 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:34191
--local_ip_for_outbound_sockets=127.0.212.66
--tserver_master_addrs=127.0.212.126:46717
--webserver_port=44735
--webserver_interface=127.0.212.66
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:54:14.388782 5030 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:54:14.389257 5030 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:14.390430 5030 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:54:14.395558 4842 ts_manager.cc:194] Registered new tserver with Master: 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65:37803)
I20250809 07:54:14.400384 4842 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 10 to 11, 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "55e9bad3aec44a45a7ad11b185083bda" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
I20250809 07:54:14.402802 4842 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) added, VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) added, VOTER 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } } }
I20250809 07:54:14.463591 4921 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:14.464542 4921 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Bootstrap complete.
I20250809 07:54:14.465999 4921 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Time spent bootstrapping tablet: real 0.151s user 0.126s sys 0.019s
I20250809 07:54:14.468246 4921 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:14.468916 4921 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Initialized, Role: FOLLOWER
I20250809 07:54:14.469535 4921 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:14.471307 4921 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250809 07:54:14.477926 4842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:40155
I20250809 07:54:14.482277 5030 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
W20250809 07:54:14.699183 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:14.699684 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:14.700201 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:14.730675 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:14.731500 5035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:54:14.766340 5035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:34191
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=44735
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:14.767585 5035 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:14.769290 5035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:14.780339 5045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:15.836716 5051 raft_consensus.cc:491] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:54:15.837369 5051 raft_consensus.cc:513] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:15.840807 5051 leader_election.cc:290] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191), 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
W20250809 07:54:15.844090 4916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.212.66:34191: connect: Connection refused (error 111)
W20250809 07:54:15.848863 4916 leader_election.cc:336] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191): Network error: Client connection negotiation failed: client connection to 127.0.212.66:34191: connect: Connection refused (error 111)
W20250809 07:54:15.849532 4917 leader_election.cc:336] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111)
I20250809 07:54:15.849853 4917 leader_election.cc:304] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 928ba1f38b404bedbf94fa16a79809a2; no voters: 55e9bad3aec44a45a7ad11b185083bda, 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:15.850526 5051 raft_consensus.cc:2747] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250809 07:54:14.781267 5046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:15.858300 5048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:15.860459 5047 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1075 milliseconds
I20250809 07:54:15.860558 5035 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:15.861661 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:15.863658 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:15.865012 5035 hybrid_clock.cc:648] HybridClock initialized: now 1754726055864969 us; error 44 us; skew 500 ppm
I20250809 07:54:15.865752 5035 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:15.871681 5035 webserver.cc:489] Webserver started at http://127.0.212.66:44735/ using document root <none> and password file <none>
I20250809 07:54:15.872579 5035 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:15.872833 5035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:15.880103 5035 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.007s sys 0.001s
I20250809 07:54:15.884423 5059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:15.885316 5035 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250809 07:54:15.885605 5035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "55e9bad3aec44a45a7ad11b185083bda"
format_stamp: "Formatted at 2025-08-09 07:53:47 on dist-test-slave-tkp6"
I20250809 07:54:15.887610 5035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:15.949160 5035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:15.950557 5035 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:15.950959 5035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:15.953333 5035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:15.958582 5066 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250809 07:54:15.969530 5035 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250809 07:54:15.969779 5035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20250809 07:54:15.970031 5035 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250809 07:54:15.974980 5066 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap starting.
I20250809 07:54:15.977501 5035 ts_tablet_manager.cc:610] Registered 2 tablets
I20250809 07:54:15.977698 5035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250809 07:54:16.022884 5066 log.cc:826] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:16.105676 5066 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:16.106580 5066 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap complete.
I20250809 07:54:16.107818 5066 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Time spent bootstrapping tablet: real 0.133s user 0.104s sys 0.028s
I20250809 07:54:16.123977 5066 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.126505 5066 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Initialized, Role: FOLLOWER
I20250809 07:54:16.127628 5066 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.130934 5066 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda: Time spent starting tablet: real 0.023s user 0.022s sys 0.000s
I20250809 07:54:16.131518 5066 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap starting.
I20250809 07:54:16.137053 5035 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:34191
I20250809 07:54:16.137321 5174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:34191 every 8 connection(s)
I20250809 07:54:16.139475 5035 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:54:16.144627 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5035
I20250809 07:54:16.146363 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:39695
--local_ip_for_outbound_sockets=127.0.212.67
--tserver_master_addrs=127.0.212.126:46717
--webserver_port=44787
--webserver_interface=127.0.212.67
--builtin_ntp_servers=127.0.212.84:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 07:54:16.171602 5175 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:54:16.172133 5175 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:16.173349 5175 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:54:16.177649 4842 ts_manager.cc:194] Registered new tserver with Master: 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191)
I20250809 07:54:16.181579 4842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:37837
I20250809 07:54:16.236402 5066 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:16.236809 5051 raft_consensus.cc:491] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:54:16.237169 5066 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Bootstrap complete.
I20250809 07:54:16.237251 5051 raft_consensus.cc:513] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.238504 5066 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Time spent bootstrapping tablet: real 0.107s user 0.078s sys 0.023s
I20250809 07:54:16.239025 5051 leader_election.cc:290] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191), 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:54:16.240762 5066 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.241389 5066 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e9bad3aec44a45a7ad11b185083bda, State: Initialized, Role: FOLLOWER
I20250809 07:54:16.241997 5066 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.243811 5066 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20250809 07:54:16.243867 4917 leader_election.cc:336] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111)
I20250809 07:54:16.244567 5175 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
I20250809 07:54:16.255692 5129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" candidate_uuid: "928ba1f38b404bedbf94fa16a79809a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "55e9bad3aec44a45a7ad11b185083bda" is_pre_election: true
I20250809 07:54:16.256458 5129 raft_consensus.cc:2466] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928ba1f38b404bedbf94fa16a79809a2 in term 2.
I20250809 07:54:16.257592 4916 leader_election.cc:304] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55e9bad3aec44a45a7ad11b185083bda, 928ba1f38b404bedbf94fa16a79809a2; no voters: 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:16.258181 5051 raft_consensus.cc:2802] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250809 07:54:16.258476 5051 raft_consensus.cc:491] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:54:16.258729 5051 raft_consensus.cc:3058] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:54:16.264401 5051 raft_consensus.cc:513] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.266458 5051 leader_election.cc:290] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 election: Requested vote from peers 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191), 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:54:16.266458 5129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" candidate_uuid: "928ba1f38b404bedbf94fa16a79809a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "55e9bad3aec44a45a7ad11b185083bda"
I20250809 07:54:16.266948 5129 raft_consensus.cc:3058] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Advancing to term 3
W20250809 07:54:16.269212 4917 leader_election.cc:336] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111)
I20250809 07:54:16.273360 5129 raft_consensus.cc:2466] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928ba1f38b404bedbf94fa16a79809a2 in term 3.
I20250809 07:54:16.274080 4916 leader_election.cc:304] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55e9bad3aec44a45a7ad11b185083bda, 928ba1f38b404bedbf94fa16a79809a2; no voters: 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:16.274601 5051 raft_consensus.cc:2802] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 FOLLOWER]: Leader election won for term 3
I20250809 07:54:16.276319 5051 raft_consensus.cc:695] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 LEADER]: Becoming Leader. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Running, Role: LEADER
I20250809 07:54:16.277101 5051 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:16.288700 4842 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: term changed from 2 to 3, leader changed from <none> to 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65). New cstate: current_term: 3 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250809 07:54:16.472179 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:16.472628 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:16.473055 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:16.503305 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:16.504108 5179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:54:16.537557 5179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:39695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=44787
--tserver_master_addrs=127.0.212.126:46717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:16.538789 5179 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:16.540261 5179 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:16.551574 5193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:16.707978 4917 consensus_peers.cc:489] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 -> Peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Couldn't send request to peer 848d63026bfa48a58b9c402bc70bad0f. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250809 07:54:16.822052 5129 raft_consensus.cc:1273] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250809 07:54:16.824406 5051 consensus_queue.cc:1035] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250809 07:54:16.931389 4984 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } }
I20250809 07:54:16.935650 5129 raft_consensus.cc:1273] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250809 07:54:16.937363 5201 consensus_queue.cc:1035] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250809 07:54:16.944525 5201 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } }
I20250809 07:54:16.948434 5129 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } }
I20250809 07:54:16.969624 4829 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 07:54:16.975178 4842 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 12 to 14, VOTER 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67) evicted. New cstate: current_term: 3 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250809 07:54:17.009209 4842 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 on TS 848d63026bfa48a58b9c402bc70bad0f: Not found: failed to reset TS proxy: Could not find TS for UUID 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:17.015894 4984 consensus_queue.cc:237] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } }
I20250809 07:54:17.018709 5201 raft_consensus.cc:2953] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } }
I20250809 07:54:17.031087 4829 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 07:54:17.036284 4842 catalog_manager.cc:5582] T 3e8c3e85a4624afd99ce8b920e6aee14 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: config changed from index 14 to 15, VOTER 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) evicted. New cstate: current_term: 3 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } health_report { overall_health: HEALTHY } } }
W20250809 07:54:17.063455 4827 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 on TS 848d63026bfa48a58b9c402bc70bad0f failed: Not found: failed to reset TS proxy: Could not find TS for UUID 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:17.086566 5109 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with delete_type TABLET_DATA_TOMBSTONED (TS 55e9bad3aec44a45a7ad11b185083bda not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:34444
I20250809 07:54:17.103688 5211 tablet_replica.cc:331] stopping tablet replica
I20250809 07:54:17.104749 5211 raft_consensus.cc:2241] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Raft consensus shutting down.
I20250809 07:54:17.105531 5211 raft_consensus.cc:2270] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Raft consensus is shut down!
I20250809 07:54:17.109478 5211 ts_tablet_manager.cc:1905] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 07:54:17.125691 5211 ts_tablet_manager.cc:1918] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250809 07:54:17.126276 5211 log.cc:1199] T 3e8c3e85a4624afd99ce8b920e6aee14 P 55e9bad3aec44a45a7ad11b185083bda: Deleting WAL directory at /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/wals/3e8c3e85a4624afd99ce8b920e6aee14
I20250809 07:54:17.128361 4827 catalog_manager.cc:4928] TS 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191): tablet 3e8c3e85a4624afd99ce8b920e6aee14 (table TestTable [id=f5c17147c1ea45e283b874c44437f3dc]) successfully deleted
I20250809 07:54:17.431797 5201 raft_consensus.cc:491] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:54:17.432334 5201 raft_consensus.cc:513] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:17.434279 5201 leader_election.cc:290] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191), 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:54:17.436790 5129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdddb55a6cb141aba67217cb66682260" candidate_uuid: "928ba1f38b404bedbf94fa16a79809a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "55e9bad3aec44a45a7ad11b185083bda" is_pre_election: true
I20250809 07:54:17.437463 5129 raft_consensus.cc:2466] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928ba1f38b404bedbf94fa16a79809a2 in term 2.
I20250809 07:54:17.438907 4916 leader_election.cc:304] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e9bad3aec44a45a7ad11b185083bda, 928ba1f38b404bedbf94fa16a79809a2; no voters:
I20250809 07:54:17.439770 5201 raft_consensus.cc:2802] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250809 07:54:17.440119 5201 raft_consensus.cc:491] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:54:17.440419 5201 raft_consensus.cc:3058] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:54:17.451203 5201 raft_consensus.cc:513] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:17.453924 5129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdddb55a6cb141aba67217cb66682260" candidate_uuid: "928ba1f38b404bedbf94fa16a79809a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "55e9bad3aec44a45a7ad11b185083bda"
I20250809 07:54:17.454483 5129 raft_consensus.cc:3058] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:54:17.460947 5129 raft_consensus.cc:2466] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928ba1f38b404bedbf94fa16a79809a2 in term 3.
I20250809 07:54:17.462047 4916 leader_election.cc:304] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e9bad3aec44a45a7ad11b185083bda, 928ba1f38b404bedbf94fa16a79809a2; no voters:
I20250809 07:54:17.462888 5051 raft_consensus.cc:2802] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 FOLLOWER]: Leader election won for term 3
I20250809 07:54:17.479948 5051 raft_consensus.cc:695] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [term 3 LEADER]: Becoming Leader. State: Replica: 928ba1f38b404bedbf94fa16a79809a2, State: Running, Role: LEADER
I20250809 07:54:17.481374 5051 consensus_queue.cc:237] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:17.484012 5201 leader_election.cc:290] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 election: Requested vote from peers 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66:34191), 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:54:17.500170 4841 catalog_manager.cc:5582] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 reported cstate change: term changed from 2 to 3, leader changed from 55e9bad3aec44a45a7ad11b185083bda (127.0.212.66) to 928ba1f38b404bedbf94fa16a79809a2 (127.0.212.65). New cstate: current_term: 3 leader_uuid: "928ba1f38b404bedbf94fa16a79809a2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250809 07:54:17.517232 4917 leader_election.cc:336] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111)
W20250809 07:54:17.520603 4917 leader_election.cc:336] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111)
I20250809 07:54:17.889006 5129 raft_consensus.cc:1273] T bdddb55a6cb141aba67217cb66682260 P 55e9bad3aec44a45a7ad11b185083bda [term 3 FOLLOWER]: Refusing update from remote peer 928ba1f38b404bedbf94fa16a79809a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250809 07:54:17.890630 5051 consensus_queue.cc:1035] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250809 07:54:17.937992 4917 consensus_peers.cc:489] T bdddb55a6cb141aba67217cb66682260 P 928ba1f38b404bedbf94fa16a79809a2 -> Peer 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): Couldn't send request to peer 848d63026bfa48a58b9c402bc70bad0f. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.67:39695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250809 07:54:17.953998 5192 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 5179
W20250809 07:54:18.014700 5179 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.463s user 0.498s sys 0.932s
W20250809 07:54:18.015071 5179 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.464s user 0.499s sys 0.932s
W20250809 07:54:16.552249 5194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:18.017338 5196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:18.019447 5195 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1462 milliseconds
I20250809 07:54:18.019515 5179 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:18.020614 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:18.022553 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:18.023903 5179 hybrid_clock.cc:648] HybridClock initialized: now 1754726058023867 us; error 39 us; skew 500 ppm
I20250809 07:54:18.024629 5179 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:18.030222 5179 webserver.cc:489] Webserver started at http://127.0.212.67:44787/ using document root <none> and password file <none>
I20250809 07:54:18.031075 5179 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:18.031282 5179 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:18.038370 5179 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20250809 07:54:18.042865 5224 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:18.044015 5179 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250809 07:54:18.044369 5179 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "848d63026bfa48a58b9c402bc70bad0f"
format_stamp: "Formatted at 2025-08-09 07:53:48 on dist-test-slave-tkp6"
I20250809 07:54:18.046162 5179 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:18.096437 5179 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:18.097787 5179 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:18.098189 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:18.100556 5179 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:18.105818 5231 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250809 07:54:18.120505 5179 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250809 07:54:18.120745 5179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.001s
I20250809 07:54:18.121048 5179 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250809 07:54:18.125893 5231 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:54:18.131354 5179 ts_tablet_manager.cc:610] Registered 3 tablets
I20250809 07:54:18.131534 5179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.007s sys 0.000s
I20250809 07:54:18.177258 5231 log.cc:826] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Log is configured to *not* fsync() on all Append() calls
W20250809 07:54:18.230363 4827 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 3e8c3e85a4624afd99ce8b920e6aee14 on TS 848d63026bfa48a58b9c402bc70bad0f failed: Not found: failed to reset TS proxy: Could not find TS for UUID 848d63026bfa48a58b9c402bc70bad0f
I20250809 07:54:18.270817 5231 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:18.271525 5231 tablet_bootstrap.cc:492] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap complete.
I20250809 07:54:18.272608 5231 ts_tablet_manager.cc:1397] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.147s user 0.107s sys 0.034s
I20250809 07:54:18.287358 5231 raft_consensus.cc:357] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:18.289805 5231 raft_consensus.cc:738] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: FOLLOWER
I20250809 07:54:18.290655 5231 consensus_queue.cc:260] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } } peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:18.294519 5231 ts_tablet_manager.cc:1428] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.022s user 0.020s sys 0.002s
I20250809 07:54:18.295085 5231 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:54:18.312026 5179 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:39695
I20250809 07:54:18.312470 5339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:39695 every 8 connection(s)
I20250809 07:54:18.315549 5179 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:54:18.326591 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5179
I20250809 07:54:18.342413 5340 heartbeater.cc:344] Connected to a master server at 127.0.212.126:46717
I20250809 07:54:18.342823 5340 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:18.343700 5340 heartbeater.cc:507] Master 127.0.212.126:46717 requested a full tablet report, sending...
I20250809 07:54:18.348007 4841 ts_manager.cc:194] Registered new tserver with Master: 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695)
I20250809 07:54:18.353425 4841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:34129
I20250809 07:54:18.362645 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:54:18.366511 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250809 07:54:18.369442 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250809 07:54:18.409461 5231 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:18.410085 5231 tablet_bootstrap.cc:492] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap complete.
I20250809 07:54:18.411008 5231 ts_tablet_manager.cc:1397] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.116s user 0.092s sys 0.019s
I20250809 07:54:18.412534 5231 raft_consensus.cc:357] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:18.412968 5231 raft_consensus.cc:738] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: FOLLOWER
I20250809 07:54:18.413435 5231 consensus_queue.cc:260] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "928ba1f38b404bedbf94fa16a79809a2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37803 } } peers { permanent_uuid: "55e9bad3aec44a45a7ad11b185083bda" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 34191 } attrs { promote: false } } peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } attrs { promote: false } }
I20250809 07:54:18.414306 5340 heartbeater.cc:499] Master 127.0.212.126:46717 was elected leader, sending a full tablet report...
I20250809 07:54:18.414899 5231 ts_tablet_manager.cc:1428] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250809 07:54:18.415576 5231 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap starting.
I20250809 07:54:18.494254 5294 raft_consensus.cc:3058] T bdddb55a6cb141aba67217cb66682260 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:54:18.500801 5231 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:54:18.501499 5231 tablet_bootstrap.cc:492] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Bootstrap complete.
I20250809 07:54:18.502489 5231 ts_tablet_manager.cc:1397] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Time spent bootstrapping tablet: real 0.087s user 0.069s sys 0.015s
I20250809 07:54:18.504357 5231 raft_consensus.cc:357] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:18.504879 5231 raft_consensus.cc:738] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Initialized, Role: FOLLOWER
I20250809 07:54:18.505378 5231 consensus_queue.cc:260] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:18.505829 5231 raft_consensus.cc:397] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:54:18.506141 5231 raft_consensus.cc:491] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:54:18.506585 5231 raft_consensus.cc:3058] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Advancing to term 3
I20250809 07:54:18.510439 5231 raft_consensus.cc:513] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:18.510988 5231 leader_election.cc:304] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 848d63026bfa48a58b9c402bc70bad0f; no voters:
I20250809 07:54:18.511406 5231 leader_election.cc:290] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [CANDIDATE]: Term 3 election: Requested vote from peers
I20250809 07:54:18.511639 5327 raft_consensus.cc:2802] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 3 FOLLOWER]: Leader election won for term 3
I20250809 07:54:18.514403 5327 raft_consensus.cc:695] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [term 3 LEADER]: Becoming Leader. State: Replica: 848d63026bfa48a58b9c402bc70bad0f, State: Running, Role: LEADER
I20250809 07:54:18.514497 5231 ts_tablet_manager.cc:1428] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20250809 07:54:18.515338 5327 consensus_queue.cc:237] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } }
I20250809 07:54:18.521696 4841 catalog_manager.cc:5582] T 31b5e6b717094538b635418648976f3b P 848d63026bfa48a58b9c402bc70bad0f reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "848d63026bfa48a58b9c402bc70bad0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848d63026bfa48a58b9c402bc70bad0f" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 39695 } health_report { overall_health: HEALTHY } } }
I20250809 07:54:19.314353 5274 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e8c3e85a4624afd99ce8b920e6aee14 with delete_type TABLET_DATA_TOMBSTONED (TS 848d63026bfa48a58b9c402bc70bad0f not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:40670
I20250809 07:54:19.319083 5363 tablet_replica.cc:331] stopping tablet replica
I20250809 07:54:19.319692 5363 raft_consensus.cc:2241] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250809 07:54:19.320139 5363 raft_consensus.cc:2270] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250809 07:54:19.322628 5363 ts_tablet_manager.cc:1905] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 07:54:19.336856 5363 ts_tablet_manager.cc:1918] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250809 07:54:19.337258 5363 log.cc:1199] T 3e8c3e85a4624afd99ce8b920e6aee14 P 848d63026bfa48a58b9c402bc70bad0f: Deleting WAL directory at /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/wals/3e8c3e85a4624afd99ce8b920e6aee14
I20250809 07:54:19.338800 4828 catalog_manager.cc:4928] TS 848d63026bfa48a58b9c402bc70bad0f (127.0.212.67:39695): tablet 3e8c3e85a4624afd99ce8b920e6aee14 (table TestTable [id=f5c17147c1ea45e283b874c44437f3dc]) successfully deleted
W20250809 07:54:19.372929 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:20.376114 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:21.380064 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:22.383426 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:23.386533 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:24.389621 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:25.392834 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:26.396028 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:27.399277 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:28.402559 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:29.405843 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:30.409060 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:31.412734 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:32.416217 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:33.419309 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:34.293987 5026 debug-util.cc:398] Leaking SignalData structure 0x7b08000b74c0 after lost signal to thread 4899
W20250809 07:54:34.294788 5026 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf9c0 after lost signal to thread 5029
W20250809 07:54:34.422751 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:35.247501 4873 debug-util.cc:398] Leaking SignalData structure 0x7b080006f600 after lost signal to thread 4810
W20250809 07:54:35.248176 4873 debug-util.cc:398] Leaking SignalData structure 0x7b080006f4c0 after lost signal to thread 4876
W20250809 07:54:35.426681 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:36.430131 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 07:54:37.433313 849 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3e8c3e85a4624afd99ce8b920e6aee14: tablet_id: "3e8c3e85a4624afd99ce8b920e6aee14" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250809 07:54:38.436089 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 4879
I20250809 07:54:38.458576 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5035
I20250809 07:54:38.478335 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5179
I20250809 07:54:38.500669 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 4808
2025-08-09T07:54:38Z chronyd exiting
I20250809 07:54:38.544417 849 test_util.cc:183] -----------------------------------------------
I20250809 07:54:38.544622 849 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754725960294695-849-0
[ FAILED ] AdminCliTest.TestRebuildTables (56868 ms)
[----------] 5 tests from AdminCliTest (118183 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250809 07:54:38.548133 849 test_util.cc:276] Using random seed: 9898127
I20250809 07:54:38.552158 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:54:38.552301 849 ts_itest-base.cc:116] --------------
I20250809 07:54:38.552412 849 ts_itest-base.cc:117] 5 tablet servers
I20250809 07:54:38.552526 849 ts_itest-base.cc:118] 3 replicas per TS
I20250809 07:54:38.552627 849 ts_itest-base.cc:119] --------------
2025-08-09T07:54:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:54:38Z Disabled control of system clock
I20250809 07:54:38.588886 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:37307
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:37307
--raft_prepare_replacement_before_eviction=true with env {}
W20250809 07:54:38.878149 5382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:38.878683 5382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:38.879078 5382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:38.908969 5382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 07:54:38.909373 5382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:54:38.909641 5382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:38.909879 5382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:54:38.910108 5382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:54:38.944278 5382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:37307
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:37307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:38.945552 5382 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:38.947053 5382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:38.957338 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:38.958463 5389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:38.961289 5391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:40.055001 5390 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1094 milliseconds
I20250809 07:54:40.055102 5382 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:40.056294 5382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:40.059042 5382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:40.060420 5382 hybrid_clock.cc:648] HybridClock initialized: now 1754726080060386 us; error 52 us; skew 500 ppm
I20250809 07:54:40.061182 5382 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:40.067106 5382 webserver.cc:489] Webserver started at http://127.0.212.126:38803/ using document root <none> and password file <none>
I20250809 07:54:40.067994 5382 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:40.068195 5382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:40.068622 5382 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:40.072691 5382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0a5a1a851135477e89be7de770a6ae3f"
format_stamp: "Formatted at 2025-08-09 07:54:40 on dist-test-slave-tkp6"
I20250809 07:54:40.073719 5382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0a5a1a851135477e89be7de770a6ae3f"
format_stamp: "Formatted at 2025-08-09 07:54:40 on dist-test-slave-tkp6"
I20250809 07:54:40.080304 5382 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250809 07:54:40.085306 5398 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:40.086252 5382 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250809 07:54:40.086539 5382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0a5a1a851135477e89be7de770a6ae3f"
format_stamp: "Formatted at 2025-08-09 07:54:40 on dist-test-slave-tkp6"
I20250809 07:54:40.086850 5382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:40.153544 5382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:40.154914 5382 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:40.155321 5382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:40.220391 5382 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:37307
I20250809 07:54:40.220445 5449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:37307 every 8 connection(s)
I20250809 07:54:40.222947 5382 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:54:40.227572 5450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:40.231770 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5382
I20250809 07:54:40.232175 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:54:40.250185 5450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f: Bootstrap starting.
I20250809 07:54:40.255374 5450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f: Neither blocks nor log segments found. Creating new log.
I20250809 07:54:40.257467 5450 log.cc:826] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:40.261742 5450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f: No bootstrap required, opened a new log
I20250809 07:54:40.277431 5450 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5a1a851135477e89be7de770a6ae3f" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37307 } }
I20250809 07:54:40.278014 5450 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:54:40.278231 5450 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5a1a851135477e89be7de770a6ae3f, State: Initialized, Role: FOLLOWER
I20250809 07:54:40.278807 5450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5a1a851135477e89be7de770a6ae3f" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37307 } }
I20250809 07:54:40.279281 5450 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:54:40.279516 5450 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:54:40.279827 5450 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:54:40.283488 5450 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5a1a851135477e89be7de770a6ae3f" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37307 } }
I20250809 07:54:40.284158 5450 leader_election.cc:304] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a5a1a851135477e89be7de770a6ae3f; no voters:
I20250809 07:54:40.285789 5450 leader_election.cc:290] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:54:40.286665 5455 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:54:40.288620 5455 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [term 1 LEADER]: Becoming Leader. State: Replica: 0a5a1a851135477e89be7de770a6ae3f, State: Running, Role: LEADER
I20250809 07:54:40.289229 5455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5a1a851135477e89be7de770a6ae3f" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37307 } }
I20250809 07:54:40.289530 5450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:54:40.299407 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a5a1a851135477e89be7de770a6ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5a1a851135477e89be7de770a6ae3f" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37307 } } }
I20250809 07:54:40.299837 5457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a5a1a851135477e89be7de770a6ae3f. Latest consensus state: current_term: 1 leader_uuid: "0a5a1a851135477e89be7de770a6ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5a1a851135477e89be7de770a6ae3f" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37307 } } }
I20250809 07:54:40.300225 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [sys.catalog]: This master's current role is: LEADER
I20250809 07:54:40.300470 5457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f [sys.catalog]: This master's current role is: LEADER
I20250809 07:54:40.302843 5465 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:54:40.313025 5465 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:54:40.326345 5465 catalog_manager.cc:1349] Generated new cluster ID: fedaa2a41d64445cb85d15d1754f93a6
I20250809 07:54:40.326630 5465 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:54:40.371649 5465 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:54:40.373173 5465 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:54:40.387594 5465 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0a5a1a851135477e89be7de770a6ae3f: Generated new TSK 0
I20250809 07:54:40.388451 5465 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:54:40.411311 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--builtin_ntp_servers=127.0.212.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250809 07:54:40.696041 5474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:40.696498 5474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:40.696974 5474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:40.727358 5474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 07:54:40.727756 5474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:40.728519 5474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:54:40.762699 5474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:40.764147 5474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:40.765630 5474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:40.777315 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:40.781908 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:40.778745 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:42.145771 5483 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1362 milliseconds
I20250809 07:54:42.145872 5474 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:42.147015 5474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:42.149524 5474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:42.150949 5474 hybrid_clock.cc:648] HybridClock initialized: now 1754726082150904 us; error 59 us; skew 500 ppm
I20250809 07:54:42.151698 5474 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:42.157915 5474 webserver.cc:489] Webserver started at http://127.0.212.65:37667/ using document root <none> and password file <none>
I20250809 07:54:42.158759 5474 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:42.158969 5474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:42.159411 5474 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:42.163513 5474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1bf1b812933f40eb98bede2466b21507"
format_stamp: "Formatted at 2025-08-09 07:54:42 on dist-test-slave-tkp6"
I20250809 07:54:42.164604 5474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1bf1b812933f40eb98bede2466b21507"
format_stamp: "Formatted at 2025-08-09 07:54:42 on dist-test-slave-tkp6"
I20250809 07:54:42.171878 5474 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 07:54:42.177508 5491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:42.178774 5474 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250809 07:54:42.179092 5474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1bf1b812933f40eb98bede2466b21507"
format_stamp: "Formatted at 2025-08-09 07:54:42 on dist-test-slave-tkp6"
I20250809 07:54:42.179397 5474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:42.236230 5474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:42.237630 5474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:42.238044 5474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:42.240434 5474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:42.244364 5474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:54:42.244560 5474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:42.244805 5474 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:54:42.244951 5474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:42.367856 5474 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:37833
I20250809 07:54:42.367951 5603 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:37833 every 8 connection(s)
I20250809 07:54:42.370260 5474 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:54:42.379040 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5474
I20250809 07:54:42.379662 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:54:42.386827 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--builtin_ntp_servers=127.0.212.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 07:54:42.393074 5604 heartbeater.cc:344] Connected to a master server at 127.0.212.126:37307
I20250809 07:54:42.393465 5604 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:42.394411 5604 heartbeater.cc:507] Master 127.0.212.126:37307 requested a full tablet report, sending...
I20250809 07:54:42.396891 5415 ts_manager.cc:194] Registered new tserver with Master: 1bf1b812933f40eb98bede2466b21507 (127.0.212.65:37833)
I20250809 07:54:42.399325 5415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:54725
W20250809 07:54:42.673477 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:42.673914 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:42.674356 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:42.704177 5608 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 07:54:42.704514 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:42.705242 5608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:54:42.738050 5608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:42.739218 5608 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:42.740676 5608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:42.751585 5614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:43.403041 5604 heartbeater.cc:499] Master 127.0.212.126:37307 was elected leader, sending a full tablet report...
W20250809 07:54:42.752254 5615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:43.862370 5617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:43.865599 5608 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.114s user 0.396s sys 0.711s
W20250809 07:54:43.866045 5608 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.115s user 0.397s sys 0.712s
W20250809 07:54:43.866988 5616 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1114 milliseconds
I20250809 07:54:43.867044 5608 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:43.868221 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:43.870352 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:43.871681 5608 hybrid_clock.cc:648] HybridClock initialized: now 1754726083871652 us; error 40 us; skew 500 ppm
I20250809 07:54:43.872473 5608 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:43.879194 5608 webserver.cc:489] Webserver started at http://127.0.212.66:38771/ using document root <none> and password file <none>
I20250809 07:54:43.880169 5608 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:43.880384 5608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:43.880801 5608 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:43.884958 5608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "00fb76eb003a4f93a5a83d89f01f72b7"
format_stamp: "Formatted at 2025-08-09 07:54:43 on dist-test-slave-tkp6"
I20250809 07:54:43.886010 5608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "00fb76eb003a4f93a5a83d89f01f72b7"
format_stamp: "Formatted at 2025-08-09 07:54:43 on dist-test-slave-tkp6"
I20250809 07:54:43.893503 5608 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 07:54:43.899214 5624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:43.900313 5608 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250809 07:54:43.900609 5608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "00fb76eb003a4f93a5a83d89f01f72b7"
format_stamp: "Formatted at 2025-08-09 07:54:43 on dist-test-slave-tkp6"
I20250809 07:54:43.900918 5608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:43.972911 5608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:43.974387 5608 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:43.974804 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:43.977214 5608 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:43.981184 5608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:54:43.981454 5608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:43.981716 5608 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:54:43.981861 5608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:44.104013 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:40677
I20250809 07:54:44.104120 5736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:40677 every 8 connection(s)
I20250809 07:54:44.106531 5608 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 07:54:44.110090 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5608
I20250809 07:54:44.110497 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 07:54:44.116539 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--builtin_ntp_servers=127.0.212.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 07:54:44.126736 5737 heartbeater.cc:344] Connected to a master server at 127.0.212.126:37307
I20250809 07:54:44.127146 5737 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:44.128173 5737 heartbeater.cc:507] Master 127.0.212.126:37307 requested a full tablet report, sending...
I20250809 07:54:44.130242 5415 ts_manager.cc:194] Registered new tserver with Master: 00fb76eb003a4f93a5a83d89f01f72b7 (127.0.212.66:40677)
I20250809 07:54:44.131824 5415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:35833
W20250809 07:54:44.409933 5741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:44.410560 5741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:44.411091 5741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:44.441867 5741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 07:54:44.442232 5741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:44.442937 5741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:54:44.476258 5741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:44.477442 5741 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:44.478909 5741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:44.489987 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:45.134768 5737 heartbeater.cc:499] Master 127.0.212.126:37307 was elected leader, sending a full tablet report...
W20250809 07:54:44.490381 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:45.593309 5749 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1098 milliseconds
W20250809 07:54:45.594442 5741 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.104s user 0.397s sys 0.701s
W20250809 07:54:45.594696 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:45.594841 5741 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.105s user 0.397s sys 0.701s
I20250809 07:54:45.595151 5741 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:45.596190 5741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:45.598273 5741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:45.599606 5741 hybrid_clock.cc:648] HybridClock initialized: now 1754726085599557 us; error 56 us; skew 500 ppm
I20250809 07:54:45.600466 5741 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:45.607355 5741 webserver.cc:489] Webserver started at http://127.0.212.67:40573/ using document root <none> and password file <none>
I20250809 07:54:45.608338 5741 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:45.608537 5741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:45.608968 5741 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:45.613196 5741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "44842f10e31c460280e9ccb2292d2f5c"
format_stamp: "Formatted at 2025-08-09 07:54:45 on dist-test-slave-tkp6"
I20250809 07:54:45.614214 5741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "44842f10e31c460280e9ccb2292d2f5c"
format_stamp: "Formatted at 2025-08-09 07:54:45 on dist-test-slave-tkp6"
I20250809 07:54:45.621699 5741 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 07:54:45.628858 5757 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:45.629976 5741 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250809 07:54:45.630285 5741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "44842f10e31c460280e9ccb2292d2f5c"
format_stamp: "Formatted at 2025-08-09 07:54:45 on dist-test-slave-tkp6"
I20250809 07:54:45.630620 5741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:45.700104 5741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:45.701545 5741 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:45.701951 5741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:45.704396 5741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:45.708302 5741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:54:45.708490 5741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:45.708726 5741 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:54:45.708868 5741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:45.832216 5741 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:37825
I20250809 07:54:45.832309 5869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:37825 every 8 connection(s)
I20250809 07:54:45.834641 5741 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 07:54:45.843681 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5741
I20250809 07:54:45.844071 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 07:54:45.849870 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.68:0
--local_ip_for_outbound_sockets=127.0.212.68
--webserver_interface=127.0.212.68
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--builtin_ntp_servers=127.0.212.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 07:54:45.858208 5870 heartbeater.cc:344] Connected to a master server at 127.0.212.126:37307
I20250809 07:54:45.858608 5870 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:45.859578 5870 heartbeater.cc:507] Master 127.0.212.126:37307 requested a full tablet report, sending...
I20250809 07:54:45.861605 5415 ts_manager.cc:194] Registered new tserver with Master: 44842f10e31c460280e9ccb2292d2f5c (127.0.212.67:37825)
I20250809 07:54:45.862794 5415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:57005
W20250809 07:54:46.135571 5874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:46.136093 5874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:46.136585 5874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:46.167375 5874 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 07:54:46.167810 5874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:46.168588 5874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.68
I20250809 07:54:46.202808 5874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.212.68
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.68
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:46.204165 5874 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:46.205683 5874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:46.217207 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:46.865736 5870 heartbeater.cc:499] Master 127.0.212.126:37307 was elected leader, sending a full tablet report...
W20250809 07:54:46.218746 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:46.224634 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:47.309966 5882 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1088 milliseconds
I20250809 07:54:47.310062 5874 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:47.311180 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:47.313266 5874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:47.314597 5874 hybrid_clock.cc:648] HybridClock initialized: now 1754726087314561 us; error 50 us; skew 500 ppm
I20250809 07:54:47.315328 5874 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:47.321386 5874 webserver.cc:489] Webserver started at http://127.0.212.68:39709/ using document root <none> and password file <none>
I20250809 07:54:47.322247 5874 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:47.322451 5874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:47.322897 5874 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:47.327143 5874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e1d693c081774966a44c06c325c3af9e"
format_stamp: "Formatted at 2025-08-09 07:54:47 on dist-test-slave-tkp6"
I20250809 07:54:47.328228 5874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e1d693c081774966a44c06c325c3af9e"
format_stamp: "Formatted at 2025-08-09 07:54:47 on dist-test-slave-tkp6"
I20250809 07:54:47.334805 5874 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250809 07:54:47.340433 5890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:47.341440 5874 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250809 07:54:47.341727 5874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e1d693c081774966a44c06c325c3af9e"
format_stamp: "Formatted at 2025-08-09 07:54:47 on dist-test-slave-tkp6"
I20250809 07:54:47.342024 5874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:47.384609 5874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:47.386129 5874 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:47.386641 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:47.389006 5874 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:47.392822 5874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:54:47.393005 5874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:47.393191 5874 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:54:47.393322 5874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:47.515486 5874 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.68:37517
I20250809 07:54:47.515586 6002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.68:37517 every 8 connection(s)
I20250809 07:54:47.517992 5874 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250809 07:54:47.523052 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 5874
I20250809 07:54:47.523561 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250809 07:54:47.529757 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.69:0
--local_ip_for_outbound_sockets=127.0.212.69
--webserver_interface=127.0.212.69
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--builtin_ntp_servers=127.0.212.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 07:54:47.537926 6003 heartbeater.cc:344] Connected to a master server at 127.0.212.126:37307
I20250809 07:54:47.538403 6003 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:47.539603 6003 heartbeater.cc:507] Master 127.0.212.126:37307 requested a full tablet report, sending...
I20250809 07:54:47.541716 5415 ts_manager.cc:194] Registered new tserver with Master: e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517)
I20250809 07:54:47.542870 5415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.68:37363
W20250809 07:54:47.819067 6007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:47.819526 6007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:47.820045 6007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:47.857482 6007 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 07:54:47.857867 6007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:47.858605 6007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.69
I20250809 07:54:47.891991 6007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.212.69
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.69
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:47.893193 6007 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:47.894661 6007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:47.905527 6013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:48.546139 6003 heartbeater.cc:499] Master 127.0.212.126:37307 was elected leader, sending a full tablet report...
W20250809 07:54:49.309129 6012 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 6007
W20250809 07:54:49.412880 6007 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.507s user 0.610s sys 0.896s
W20250809 07:54:49.413230 6007 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.508s user 0.610s sys 0.896s
W20250809 07:54:47.906394 6014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:49.414034 6015 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1507 milliseconds
W20250809 07:54:49.415490 6016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:54:49.415421 6007 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:49.418740 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:49.420683 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:49.422058 6007 hybrid_clock.cc:648] HybridClock initialized: now 1754726089422012 us; error 42 us; skew 500 ppm
I20250809 07:54:49.422758 6007 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:49.428249 6007 webserver.cc:489] Webserver started at http://127.0.212.69:42675/ using document root <none> and password file <none>
I20250809 07:54:49.429217 6007 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:49.429425 6007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:49.429895 6007 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:49.434062 6007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "378c6f87968e415fbbbde93d99c1840b"
format_stamp: "Formatted at 2025-08-09 07:54:49 on dist-test-slave-tkp6"
I20250809 07:54:49.435034 6007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "378c6f87968e415fbbbde93d99c1840b"
format_stamp: "Formatted at 2025-08-09 07:54:49 on dist-test-slave-tkp6"
I20250809 07:54:49.441474 6007 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250809 07:54:49.446498 6024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:49.447397 6007 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250809 07:54:49.447685 6007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "378c6f87968e415fbbbde93d99c1840b"
format_stamp: "Formatted at 2025-08-09 07:54:49 on dist-test-slave-tkp6"
I20250809 07:54:49.448011 6007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:49.507926 6007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:49.509317 6007 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:49.509711 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:49.512172 6007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:54:49.516495 6007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:54:49.516707 6007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:49.516985 6007 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:54:49.517197 6007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:49.660140 6007 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.69:40645
I20250809 07:54:49.660236 6136 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.69:40645 every 8 connection(s)
I20250809 07:54:49.662482 6007 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250809 07:54:49.665675 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6007
I20250809 07:54:49.666175 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754725960294695-849-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250809 07:54:49.684509 6137 heartbeater.cc:344] Connected to a master server at 127.0.212.126:37307
I20250809 07:54:49.684937 6137 heartbeater.cc:461] Registering TS with master...
I20250809 07:54:49.685989 6137 heartbeater.cc:507] Master 127.0.212.126:37307 requested a full tablet report, sending...
I20250809 07:54:49.688181 5414 ts_manager.cc:194] Registered new tserver with Master: 378c6f87968e415fbbbde93d99c1840b (127.0.212.69:40645)
I20250809 07:54:49.689605 5414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.69:51767
I20250809 07:54:49.700762 849 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250809 07:54:49.735917 5414 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41374:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250809 07:54:49.809095 5938 tablet_service.cc:1468] Processing CreateTablet for tablet 3efab90e90c444de998748f9a29b4ed5 (DEFAULT_TABLE table=TestTable [id=d8d3c1fabae0465fae91dce66dca15a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:54:49.811717 5938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efab90e90c444de998748f9a29b4ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:49.818912 5672 tablet_service.cc:1468] Processing CreateTablet for tablet 3efab90e90c444de998748f9a29b4ed5 (DEFAULT_TABLE table=TestTable [id=d8d3c1fabae0465fae91dce66dca15a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:54:49.820966 5672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efab90e90c444de998748f9a29b4ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:49.820780 5539 tablet_service.cc:1468] Processing CreateTablet for tablet 3efab90e90c444de998748f9a29b4ed5 (DEFAULT_TABLE table=TestTable [id=d8d3c1fabae0465fae91dce66dca15a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:54:49.822997 5539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efab90e90c444de998748f9a29b4ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:49.848969 6157 tablet_bootstrap.cc:492] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: Bootstrap starting.
I20250809 07:54:49.850220 6156 tablet_bootstrap.cc:492] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: Bootstrap starting.
I20250809 07:54:49.852519 6158 tablet_bootstrap.cc:492] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: Bootstrap starting.
I20250809 07:54:49.856853 6157 tablet_bootstrap.cc:654] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: Neither blocks nor log segments found. Creating new log.
I20250809 07:54:49.858199 6156 tablet_bootstrap.cc:654] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: Neither blocks nor log segments found. Creating new log.
I20250809 07:54:49.859020 6158 tablet_bootstrap.cc:654] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: Neither blocks nor log segments found. Creating new log.
I20250809 07:54:49.858994 6157 log.cc:826] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:49.860698 6158 log.cc:826] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:49.860821 6156 log.cc:826] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:49.864991 6157 tablet_bootstrap.cc:492] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: No bootstrap required, opened a new log
I20250809 07:54:49.865458 6157 ts_tablet_manager.cc:1397] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250809 07:54:49.866847 6158 tablet_bootstrap.cc:492] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: No bootstrap required, opened a new log
I20250809 07:54:49.867297 6156 tablet_bootstrap.cc:492] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: No bootstrap required, opened a new log
I20250809 07:54:49.867442 6158 ts_tablet_manager.cc:1397] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20250809 07:54:49.867668 6156 ts_tablet_manager.cc:1397] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250809 07:54:49.885839 6156 raft_consensus.cc:357] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.886499 6156 raft_consensus.cc:383] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:54:49.886758 6156 raft_consensus.cc:738] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d693c081774966a44c06c325c3af9e, State: Initialized, Role: FOLLOWER
I20250809 07:54:49.887513 6156 consensus_queue.cc:260] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.891834 6156 ts_tablet_manager.cc:1428] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: Time spent starting tablet: real 0.024s user 0.018s sys 0.006s
I20250809 07:54:49.891947 6157 raft_consensus.cc:357] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.893013 6157 raft_consensus.cc:383] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:54:49.893335 6157 raft_consensus.cc:738] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fb76eb003a4f93a5a83d89f01f72b7, State: Initialized, Role: FOLLOWER
I20250809 07:54:49.892375 6158 raft_consensus.cc:357] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.893695 6158 raft_consensus.cc:383] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:54:49.894004 6158 raft_consensus.cc:738] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf1b812933f40eb98bede2466b21507, State: Initialized, Role: FOLLOWER
I20250809 07:54:49.894621 6157 consensus_queue.cc:260] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.894819 6158 consensus_queue.cc:260] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.901852 6157 ts_tablet_manager.cc:1428] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: Time spent starting tablet: real 0.036s user 0.028s sys 0.008s
I20250809 07:54:49.904799 6158 ts_tablet_manager.cc:1428] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: Time spent starting tablet: real 0.037s user 0.029s sys 0.010s
I20250809 07:54:49.966776 6163 raft_consensus.cc:491] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:54:49.967247 6163 raft_consensus.cc:513] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.970037 6163 leader_election.cc:290] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517), 00fb76eb003a4f93a5a83d89f01f72b7 (127.0.212.66:40677)
I20250809 07:54:49.981084 5958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3efab90e90c444de998748f9a29b4ed5" candidate_uuid: "1bf1b812933f40eb98bede2466b21507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d693c081774966a44c06c325c3af9e" is_pre_election: true
I20250809 07:54:49.981950 5958 raft_consensus.cc:2466] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf1b812933f40eb98bede2466b21507 in term 0.
I20250809 07:54:49.983120 5492 leader_election.cc:304] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf1b812933f40eb98bede2466b21507, e1d693c081774966a44c06c325c3af9e; no voters:
I20250809 07:54:49.983552 5692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3efab90e90c444de998748f9a29b4ed5" candidate_uuid: "1bf1b812933f40eb98bede2466b21507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" is_pre_election: true
I20250809 07:54:49.984244 6163 raft_consensus.cc:2802] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:54:49.984567 6163 raft_consensus.cc:491] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:54:49.984733 5692 raft_consensus.cc:2466] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf1b812933f40eb98bede2466b21507 in term 0.
I20250809 07:54:49.984869 6163 raft_consensus.cc:3058] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:54:49.990115 6163 raft_consensus.cc:513] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:49.991483 6163 leader_election.cc:290] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [CANDIDATE]: Term 1 election: Requested vote from peers e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517), 00fb76eb003a4f93a5a83d89f01f72b7 (127.0.212.66:40677)
I20250809 07:54:49.992085 5958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3efab90e90c444de998748f9a29b4ed5" candidate_uuid: "1bf1b812933f40eb98bede2466b21507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d693c081774966a44c06c325c3af9e"
I20250809 07:54:49.992350 5692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3efab90e90c444de998748f9a29b4ed5" candidate_uuid: "1bf1b812933f40eb98bede2466b21507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fb76eb003a4f93a5a83d89f01f72b7"
I20250809 07:54:49.992491 5958 raft_consensus.cc:3058] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:54:49.992748 5692 raft_consensus.cc:3058] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:54:49.996932 5958 raft_consensus.cc:2466] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf1b812933f40eb98bede2466b21507 in term 1.
I20250809 07:54:49.997326 5692 raft_consensus.cc:2466] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf1b812933f40eb98bede2466b21507 in term 1.
I20250809 07:54:49.997653 5492 leader_election.cc:304] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf1b812933f40eb98bede2466b21507, e1d693c081774966a44c06c325c3af9e; no voters:
I20250809 07:54:49.998245 6163 raft_consensus.cc:2802] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:54:50.000190 6163 raft_consensus.cc:695] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [term 1 LEADER]: Becoming Leader. State: Replica: 1bf1b812933f40eb98bede2466b21507, State: Running, Role: LEADER
I20250809 07:54:50.000952 6163 consensus_queue.cc:237] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } }
I20250809 07:54:50.012110 5414 catalog_manager.cc:5582] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bf1b812933f40eb98bede2466b21507 (127.0.212.65). New cstate: current_term: 1 leader_uuid: "1bf1b812933f40eb98bede2466b21507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d693c081774966a44c06c325c3af9e" member_type: VOTER last_known_addr { host: "127.0.212.68" port: 37517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf1b812933f40eb98bede2466b21507" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 37833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 } health_report { overall_health: UNKNOWN } } }
W20250809 07:54:50.024068 6004 tablet.cc:2378] T 3efab90e90c444de998748f9a29b4ed5 P e1d693c081774966a44c06c325c3af9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:54:50.042272 849 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250809 07:54:50.045318 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1bf1b812933f40eb98bede2466b21507 to finish bootstrapping
I20250809 07:54:50.057538 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 00fb76eb003a4f93a5a83d89f01f72b7 to finish bootstrapping
I20250809 07:54:50.067170 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e1d693c081774966a44c06c325c3af9e to finish bootstrapping
I20250809 07:54:50.077616 849 test_util.cc:276] Using random seed: 21427614
I20250809 07:54:50.100517 849 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250809 07:54:50.101310 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5874
W20250809 07:54:50.118844 5738 tablet.cc:2378] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:54:50.133560 5692 raft_consensus.cc:1273] T 3efab90e90c444de998748f9a29b4ed5 P 00fb76eb003a4f93a5a83d89f01f72b7 [term 1 FOLLOWER]: Refusing update from remote peer 1bf1b812933f40eb98bede2466b21507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250809 07:54:50.133981 5492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.212.68:37517: connect: Connection refused (error 111)
W20250809 07:54:50.134735 5605 tablet.cc:2378] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:54:50.139801 6167 consensus_queue.cc:1035] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fb76eb003a4f93a5a83d89f01f72b7" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 40677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250809 07:54:50.141923 5492 consensus_peers.cc:489] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 -> Peer e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517): Couldn't send request to peer e1d693c081774966a44c06c325c3af9e. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.68:37517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250809 07:54:50.155966 6177 mvcc.cc:204] Tried to move back new op lower bound from 7187358065166999552 to 7187358064650514432. Current Snapshot: MvccSnapshot[applied={T|T < 7187358065166999552}]
I20250809 07:54:50.158957 6180 mvcc.cc:204] Tried to move back new op lower bound from 7187358065166999552 to 7187358064650514432. Current Snapshot: MvccSnapshot[applied={T|T < 7187358065166999552}]
I20250809 07:54:50.694113 6137 heartbeater.cc:499] Master 127.0.212.126:37307 was elected leader, sending a full tablet report...
I20250809 07:54:52.239583 5539 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 07:54:52.240705 5672 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:54:52.251592 6072 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:54:52.293002 5805 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250809 07:54:52.573108 5492 consensus_peers.cc:489] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 -> Peer e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517): Couldn't send request to peer e1d693c081774966a44c06c325c3af9e. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.68:37517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250809 07:54:53.959278 5539 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 07:54:53.976886 6072 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:54:53.976847 5672 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:54:54.028558 5805 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250809 07:54:55.065171 5492 consensus_peers.cc:489] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 -> Peer e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517): Couldn't send request to peer e1d693c081774966a44c06c325c3af9e. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.68:37517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250809 07:54:55.529223 5492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.212.68:37517: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250809 07:54:55.684567 5539 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 07:54:55.688150 5672 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:54:55.726774 6072 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:54:55.738246 5805 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250809 07:54:57.651003 5492 consensus_peers.cc:489] T 3efab90e90c444de998748f9a29b4ed5 P 1bf1b812933f40eb98bede2466b21507 -> Peer e1d693c081774966a44c06c325c3af9e (127.0.212.68:37517): Couldn't send request to peer e1d693c081774966a44c06c325c3af9e. Status: Network error: Client connection negotiation failed: client connection to 127.0.212.68:37517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250809 07:54:58.138671 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5474
I20250809 07:54:58.171767 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5608
I20250809 07:54:58.198752 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5741
I20250809 07:54:58.220073 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6007
I20250809 07:54:58.247882 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 5382
2025-08-09T07:54:58Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19754 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19754 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250809 07:54:58.302981 849 test_util.cc:276] Using random seed: 29652972
I20250809 07:54:58.306843 849 ts_itest-base.cc:115] Starting cluster with:
I20250809 07:54:58.306999 849 ts_itest-base.cc:116] --------------
I20250809 07:54:58.307149 849 ts_itest-base.cc:117] 1 tablet servers
I20250809 07:54:58.307283 849 ts_itest-base.cc:118] 1 replicas per TS
I20250809 07:54:58.307411 849 ts_itest-base.cc:119] --------------
2025-08-09T07:54:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:54:58Z Disabled control of system clock
I20250809 07:54:58.346062 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:37665
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:37665 with env {}
W20250809 07:54:58.639022 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:54:58.639542 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:54:58.640007 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:54:58.670127 6320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:54:58.670405 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:54:58.670614 6320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:54:58.670806 6320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:54:58.704296 6320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:37665
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:54:58.705534 6320 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:54:58.707038 6320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:54:58.716856 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:58.718554 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:58.720033 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:54:59.802872 6328 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 07:54:59.802970 6320 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:54:59.806456 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:54:59.809337 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:54:59.810743 6320 hybrid_clock.cc:648] HybridClock initialized: now 1754726099810707 us; error 54 us; skew 500 ppm
I20250809 07:54:59.811522 6320 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:54:59.817538 6320 webserver.cc:489] Webserver started at http://127.0.212.126:38981/ using document root <none> and password file <none>
I20250809 07:54:59.818387 6320 fs_manager.cc:362] Metadata directory not provided
I20250809 07:54:59.818598 6320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:54:59.819041 6320 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:54:59.823369 6320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "522b3a8dcb0b41d183413d8225f7e473"
format_stamp: "Formatted at 2025-08-09 07:54:59 on dist-test-slave-tkp6"
I20250809 07:54:59.824558 6320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "522b3a8dcb0b41d183413d8225f7e473"
format_stamp: "Formatted at 2025-08-09 07:54:59 on dist-test-slave-tkp6"
I20250809 07:54:59.831233 6320 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250809 07:54:59.836400 6336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:54:59.837319 6320 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250809 07:54:59.837625 6320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "522b3a8dcb0b41d183413d8225f7e473"
format_stamp: "Formatted at 2025-08-09 07:54:59 on dist-test-slave-tkp6"
I20250809 07:54:59.837914 6320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:54:59.888366 6320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:54:59.889797 6320 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:54:59.890287 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:54:59.955585 6320 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:37665
I20250809 07:54:59.955660 6387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:37665 every 8 connection(s)
I20250809 07:54:59.958098 6320 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 07:54:59.962741 6388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:54:59.965621 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6320
I20250809 07:54:59.965971 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 07:54:59.982674 6388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473: Bootstrap starting.
I20250809 07:54:59.988647 6388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473: Neither blocks nor log segments found. Creating new log.
I20250809 07:54:59.990170 6388 log.cc:826] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473: Log is configured to *not* fsync() on all Append() calls
I20250809 07:54:59.994203 6388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473: No bootstrap required, opened a new log
I20250809 07:55:00.010860 6388 raft_consensus.cc:357] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522b3a8dcb0b41d183413d8225f7e473" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37665 } }
I20250809 07:55:00.011474 6388 raft_consensus.cc:383] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:00.011754 6388 raft_consensus.cc:738] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522b3a8dcb0b41d183413d8225f7e473, State: Initialized, Role: FOLLOWER
I20250809 07:55:00.012415 6388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522b3a8dcb0b41d183413d8225f7e473" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37665 } }
I20250809 07:55:00.012897 6388 raft_consensus.cc:397] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:55:00.013145 6388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:55:00.013437 6388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:00.017236 6388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522b3a8dcb0b41d183413d8225f7e473" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37665 } }
I20250809 07:55:00.017890 6388 leader_election.cc:304] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 522b3a8dcb0b41d183413d8225f7e473; no voters:
I20250809 07:55:00.019423 6388 leader_election.cc:290] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:55:00.020133 6393 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:55:00.022372 6393 raft_consensus.cc:695] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [term 1 LEADER]: Becoming Leader. State: Replica: 522b3a8dcb0b41d183413d8225f7e473, State: Running, Role: LEADER
I20250809 07:55:00.023175 6393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522b3a8dcb0b41d183413d8225f7e473" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37665 } }
I20250809 07:55:00.024144 6388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:55:00.033131 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 522b3a8dcb0b41d183413d8225f7e473. Latest consensus state: current_term: 1 leader_uuid: "522b3a8dcb0b41d183413d8225f7e473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522b3a8dcb0b41d183413d8225f7e473" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37665 } } }
I20250809 07:55:00.033051 6394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "522b3a8dcb0b41d183413d8225f7e473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522b3a8dcb0b41d183413d8225f7e473" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 37665 } } }
I20250809 07:55:00.033803 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:00.033829 6394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473 [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:00.041389 6402 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:55:00.049939 6402 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:55:00.063879 6402 catalog_manager.cc:1349] Generated new cluster ID: d8b9a9c457ef477093b169340d085e5a
I20250809 07:55:00.064170 6402 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:55:00.082854 6402 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:55:00.084808 6402 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:55:00.101091 6402 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 522b3a8dcb0b41d183413d8225f7e473: Generated new TSK 0
I20250809 07:55:00.101930 6402 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:55:00.118563 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37665
--builtin_ntp_servers=127.0.212.84:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 07:55:00.406196 6412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:00.406687 6412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:00.407169 6412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:00.437278 6412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:00.438078 6412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:55:00.471227 6412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:37665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:00.472522 6412 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:00.474021 6412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:00.485841 6418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:00.487835 6419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:01.863704 6420 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1375 milliseconds
W20250809 07:55:01.865494 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:01.867717 6412 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.381s user 0.485s sys 0.891s
W20250809 07:55:01.868057 6412 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.381s user 0.489s sys 0.891s
I20250809 07:55:01.868338 6412 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:01.869375 6412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:01.872360 6412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:01.873819 6412 hybrid_clock.cc:648] HybridClock initialized: now 1754726101873753 us; error 80 us; skew 500 ppm
I20250809 07:55:01.874609 6412 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:01.882146 6412 webserver.cc:489] Webserver started at http://127.0.212.65:36723/ using document root <none> and password file <none>
I20250809 07:55:01.883143 6412 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:01.883378 6412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:01.883929 6412 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:55:01.888523 6412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0c6c7b11daa344e59ef927c1641266ec"
format_stamp: "Formatted at 2025-08-09 07:55:01 on dist-test-slave-tkp6"
I20250809 07:55:01.889654 6412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0c6c7b11daa344e59ef927c1641266ec"
format_stamp: "Formatted at 2025-08-09 07:55:01 on dist-test-slave-tkp6"
I20250809 07:55:01.897300 6412 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 07:55:01.904211 6429 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:01.905620 6412 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250809 07:55:01.905984 6412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0c6c7b11daa344e59ef927c1641266ec"
format_stamp: "Formatted at 2025-08-09 07:55:01 on dist-test-slave-tkp6"
I20250809 07:55:01.906320 6412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:01.990046 6412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:01.991999 6412 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:01.992532 6412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:01.995656 6412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:02.001108 6412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:55:02.001394 6412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:02.001695 6412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:55:02.001910 6412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:02.146764 6412 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:35615
I20250809 07:55:02.146829 6541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:35615 every 8 connection(s)
I20250809 07:55:02.150600 6412 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 07:55:02.158421 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6412
I20250809 07:55:02.159147 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754725960294695-849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 07:55:02.175118 6542 heartbeater.cc:344] Connected to a master server at 127.0.212.126:37665
I20250809 07:55:02.175604 6542 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:02.176826 6542 heartbeater.cc:507] Master 127.0.212.126:37665 requested a full tablet report, sending...
I20250809 07:55:02.179359 6353 ts_manager.cc:194] Registered new tserver with Master: 0c6c7b11daa344e59ef927c1641266ec (127.0.212.65:35615)
I20250809 07:55:02.181859 6353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:52843
I20250809 07:55:02.191645 849 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250809 07:55:02.224797 6353 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56688:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250809 07:55:02.285036 6477 tablet_service.cc:1468] Processing CreateTablet for tablet cee20d2cef714efa8360a48eeb3e17d2 (DEFAULT_TABLE table=TestTable [id=eca1fb94f865400cbfc8b83abfb755ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:02.286471 6477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee20d2cef714efa8360a48eeb3e17d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:02.305351 6557 tablet_bootstrap.cc:492] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec: Bootstrap starting.
I20250809 07:55:02.310465 6557 tablet_bootstrap.cc:654] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:02.312178 6557 log.cc:826] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:02.316766 6557 tablet_bootstrap.cc:492] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec: No bootstrap required, opened a new log
I20250809 07:55:02.317129 6557 ts_tablet_manager.cc:1397] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250809 07:55:02.335222 6557 raft_consensus.cc:357] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6c7b11daa344e59ef927c1641266ec" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35615 } }
I20250809 07:55:02.335768 6557 raft_consensus.cc:383] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:02.335984 6557 raft_consensus.cc:738] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6c7b11daa344e59ef927c1641266ec, State: Initialized, Role: FOLLOWER
I20250809 07:55:02.336585 6557 consensus_queue.cc:260] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6c7b11daa344e59ef927c1641266ec" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35615 } }
I20250809 07:55:02.337046 6557 raft_consensus.cc:397] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:55:02.337291 6557 raft_consensus.cc:491] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:55:02.337574 6557 raft_consensus.cc:3058] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:02.341980 6557 raft_consensus.cc:513] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6c7b11daa344e59ef927c1641266ec" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35615 } }
I20250809 07:55:02.342641 6557 leader_election.cc:304] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6c7b11daa344e59ef927c1641266ec; no voters:
I20250809 07:55:02.344578 6557 leader_election.cc:290] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:55:02.344957 6559 raft_consensus.cc:2802] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:55:02.347883 6542 heartbeater.cc:499] Master 127.0.212.126:37665 was elected leader, sending a full tablet report...
I20250809 07:55:02.348912 6557 ts_tablet_manager.cc:1428] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec: Time spent starting tablet: real 0.032s user 0.024s sys 0.007s
I20250809 07:55:02.349411 6559 raft_consensus.cc:695] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [term 1 LEADER]: Becoming Leader. State: Replica: 0c6c7b11daa344e59ef927c1641266ec, State: Running, Role: LEADER
I20250809 07:55:02.350090 6559 consensus_queue.cc:237] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6c7b11daa344e59ef927c1641266ec" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35615 } }
I20250809 07:55:02.362068 6353 catalog_manager.cc:5582] T cee20d2cef714efa8360a48eeb3e17d2 P 0c6c7b11daa344e59ef927c1641266ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6c7b11daa344e59ef927c1641266ec (127.0.212.65). New cstate: current_term: 1 leader_uuid: "0c6c7b11daa344e59ef927c1641266ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6c7b11daa344e59ef927c1641266ec" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 35615 } health_report { overall_health: HEALTHY } } }
I20250809 07:55:02.387784 849 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250809 07:55:02.391155 849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0c6c7b11daa344e59ef927c1641266ec to finish bootstrapping
I20250809 07:55:04.959165 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6412
I20250809 07:55:04.980830 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6320
2025-08-09T07:55:05Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6727 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6727 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250809 07:55:05.030970 849 test_util.cc:276] Using random seed: 36380963
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (11 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-09T07:55:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T07:55:05Z Disabled control of system clock
I20250809 07:55:05.078271 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:33693
--webserver_interface=127.0.212.126
--webserver_port=0
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:33693 with env {}
W20250809 07:55:05.367578 6585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:05.368191 6585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:05.368598 6585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:05.399133 6585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:55:05.399430 6585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:05.399641 6585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:55:05.399869 6585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:55:05.434173 6585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:33693
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:05.435461 6585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:05.437064 6585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:05.446920 6591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:06.850056 6590 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 6585
W20250809 07:55:06.983589 6585 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.536s user 0.627s sys 0.909s
W20250809 07:55:05.448093 6592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:06.983984 6585 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.537s user 0.628s sys 0.909s
W20250809 07:55:06.985826 6594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:06.988524 6593 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1536 milliseconds
I20250809 07:55:06.988539 6585 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:06.989693 6585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:06.992018 6585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:06.993352 6585 hybrid_clock.cc:648] HybridClock initialized: now 1754726106993291 us; error 62 us; skew 500 ppm
I20250809 07:55:06.994102 6585 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:06.999688 6585 webserver.cc:489] Webserver started at http://127.0.212.126:39305/ using document root <none> and password file <none>
I20250809 07:55:07.000607 6585 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:07.000811 6585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:07.001246 6585 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:55:07.005376 6585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/instance:
uuid: "91894df50eaa4320a2a9b3c708ed8dbd"
format_stamp: "Formatted at 2025-08-09 07:55:06 on dist-test-slave-tkp6"
I20250809 07:55:07.006423 6585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal/instance:
uuid: "91894df50eaa4320a2a9b3c708ed8dbd"
format_stamp: "Formatted at 2025-08-09 07:55:06 on dist-test-slave-tkp6"
I20250809 07:55:07.013187 6585 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250809 07:55:07.018625 6601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:07.019574 6585 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250809 07:55:07.019907 6585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
uuid: "91894df50eaa4320a2a9b3c708ed8dbd"
format_stamp: "Formatted at 2025-08-09 07:55:06 on dist-test-slave-tkp6"
I20250809 07:55:07.020233 6585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:07.062620 6585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:07.064046 6585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:07.064460 6585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:07.128989 6585 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:33693
I20250809 07:55:07.129058 6652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:33693 every 8 connection(s)
I20250809 07:55:07.131572 6585 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb
I20250809 07:55:07.136301 6653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:07.140352 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6585
I20250809 07:55:07.140751 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal/instance
I20250809 07:55:07.158470 6653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd: Bootstrap starting.
I20250809 07:55:07.164436 6653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:07.165940 6653 log.cc:826] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:07.169977 6653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd: No bootstrap required, opened a new log
I20250809 07:55:07.187378 6653 raft_consensus.cc:357] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:07.188023 6653 raft_consensus.cc:383] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:07.188248 6653 raft_consensus.cc:738] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91894df50eaa4320a2a9b3c708ed8dbd, State: Initialized, Role: FOLLOWER
I20250809 07:55:07.189047 6653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:07.189692 6653 raft_consensus.cc:397] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:55:07.190013 6653 raft_consensus.cc:491] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:55:07.190405 6653 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:07.195951 6653 raft_consensus.cc:513] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:07.196759 6653 leader_election.cc:304] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91894df50eaa4320a2a9b3c708ed8dbd; no voters:
I20250809 07:55:07.198350 6653 leader_election.cc:290] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:55:07.199010 6658 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:55:07.200913 6658 raft_consensus.cc:695] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [term 1 LEADER]: Becoming Leader. State: Replica: 91894df50eaa4320a2a9b3c708ed8dbd, State: Running, Role: LEADER
I20250809 07:55:07.201650 6658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:07.202544 6653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:55:07.209623 6659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } } }
I20250809 07:55:07.209982 6660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91894df50eaa4320a2a9b3c708ed8dbd. Latest consensus state: current_term: 1 leader_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91894df50eaa4320a2a9b3c708ed8dbd" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } } }
I20250809 07:55:07.210268 6659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:07.210496 6660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:07.212838 6665 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:55:07.221468 6665 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:55:07.237116 6665 catalog_manager.cc:1349] Generated new cluster ID: 6a2d61629a114a3b8abe492bcf747121
I20250809 07:55:07.237447 6665 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:55:07.264573 6665 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:55:07.265990 6665 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:55:07.279250 6665 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 91894df50eaa4320a2a9b3c708ed8dbd: Generated new TSK 0
I20250809 07:55:07.280427 6665 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 07:55:07.300724 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:0
--local_ip_for_outbound_sockets=127.0.212.65
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:33693
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250809 07:55:07.597680 6677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:07.598179 6677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:07.598654 6677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:07.629051 6677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:07.629891 6677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:55:07.664186 6677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=0
--tserver_master_addrs=127.0.212.126:33693
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:07.665422 6677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:07.666951 6677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:07.678992 6683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:09.081187 6682 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 6677
W20250809 07:55:09.178575 6682 kernel_stack_watchdog.cc:198] Thread 6677 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 07:55:07.680680 6684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:09.179450 6677 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.500s user 0.575s sys 0.799s
W20250809 07:55:09.179900 6677 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.500s user 0.575s sys 0.799s
W20250809 07:55:09.181535 6686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:09.184355 6685 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1504 milliseconds
I20250809 07:55:09.184389 6677 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:09.185542 6677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:09.187914 6677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:09.189321 6677 hybrid_clock.cc:648] HybridClock initialized: now 1754726109189277 us; error 63 us; skew 500 ppm
I20250809 07:55:09.190068 6677 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:09.196270 6677 webserver.cc:489] Webserver started at http://127.0.212.65:44659/ using document root <none> and password file <none>
I20250809 07:55:09.197149 6677 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:09.197358 6677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:09.197773 6677 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:55:09.201850 6677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/instance:
uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2"
format_stamp: "Formatted at 2025-08-09 07:55:09 on dist-test-slave-tkp6"
I20250809 07:55:09.202914 6677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal/instance:
uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2"
format_stamp: "Formatted at 2025-08-09 07:55:09 on dist-test-slave-tkp6"
I20250809 07:55:09.209810 6677 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250809 07:55:09.215313 6693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:09.216449 6677 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250809 07:55:09.216754 6677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2"
format_stamp: "Formatted at 2025-08-09 07:55:09 on dist-test-slave-tkp6"
I20250809 07:55:09.217065 6677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:09.268388 6677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:09.269797 6677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:09.270202 6677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:09.272758 6677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:09.276804 6677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:55:09.277014 6677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:09.277237 6677 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:55:09.277385 6677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:09.425509 6677 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:34865
I20250809 07:55:09.425681 6805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:34865 every 8 connection(s)
I20250809 07:55:09.428013 6677 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb
I20250809 07:55:09.437463 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6677
I20250809 07:55:09.437863 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal/instance
I20250809 07:55:09.443488 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:0
--local_ip_for_outbound_sockets=127.0.212.66
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:33693
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 07:55:09.456604 6806 heartbeater.cc:344] Connected to a master server at 127.0.212.126:33693
I20250809 07:55:09.457010 6806 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:09.457943 6806 heartbeater.cc:507] Master 127.0.212.126:33693 requested a full tablet report, sending...
I20250809 07:55:09.460232 6618 ts_manager.cc:194] Registered new tserver with Master: 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865)
I20250809 07:55:09.462122 6618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:36851
W20250809 07:55:09.746393 6810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:09.746836 6810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:09.747256 6810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:09.776970 6810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:09.777741 6810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:55:09.810324 6810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=0
--tserver_master_addrs=127.0.212.126:33693
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:09.811475 6810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:09.812976 6810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:09.824440 6816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:55:10.465337 6806 heartbeater.cc:499] Master 127.0.212.126:33693 was elected leader, sending a full tablet report...
W20250809 07:55:09.825034 6817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:11.015554 6819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:11.018630 6818 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
W20250809 07:55:11.019347 6810 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.195s user 0.451s sys 0.732s
W20250809 07:55:11.019698 6810 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.195s user 0.451s sys 0.732s
I20250809 07:55:11.019981 6810 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:11.021333 6810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:11.023955 6810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:11.025394 6810 hybrid_clock.cc:648] HybridClock initialized: now 1754726111025342 us; error 42 us; skew 500 ppm
I20250809 07:55:11.026418 6810 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:11.034610 6810 webserver.cc:489] Webserver started at http://127.0.212.66:35393/ using document root <none> and password file <none>
I20250809 07:55:11.035916 6810 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:11.036147 6810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:11.036576 6810 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:55:11.040812 6810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/instance:
uuid: "0a4e315577314ed8a6638ed0160b2896"
format_stamp: "Formatted at 2025-08-09 07:55:11 on dist-test-slave-tkp6"
I20250809 07:55:11.041836 6810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal/instance:
uuid: "0a4e315577314ed8a6638ed0160b2896"
format_stamp: "Formatted at 2025-08-09 07:55:11 on dist-test-slave-tkp6"
I20250809 07:55:11.048978 6810 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250809 07:55:11.054494 6827 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:11.055567 6810 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250809 07:55:11.055873 6810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
uuid: "0a4e315577314ed8a6638ed0160b2896"
format_stamp: "Formatted at 2025-08-09 07:55:11 on dist-test-slave-tkp6"
I20250809 07:55:11.056171 6810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:11.129561 6810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:11.131357 6810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:11.131901 6810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:11.134982 6810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:11.140157 6810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:55:11.140425 6810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:11.140717 6810 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:55:11.140916 6810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:11.263612 6810 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:46189
I20250809 07:55:11.263685 6939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:46189 every 8 connection(s)
I20250809 07:55:11.265988 6810 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb
I20250809 07:55:11.276082 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6810
I20250809 07:55:11.276768 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal/instance
I20250809 07:55:11.283443 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:0
--local_ip_for_outbound_sockets=127.0.212.67
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:33693
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 07:55:11.286373 6940 heartbeater.cc:344] Connected to a master server at 127.0.212.126:33693
I20250809 07:55:11.286837 6940 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:11.288065 6940 heartbeater.cc:507] Master 127.0.212.126:33693 requested a full tablet report, sending...
I20250809 07:55:11.290303 6618 ts_manager.cc:194] Registered new tserver with Master: 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
I20250809 07:55:11.291486 6618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:49059
W20250809 07:55:11.572382 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:11.573004 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:11.573787 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:11.604552 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:11.605376 6944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:55:11.638548 6944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=0
--tserver_master_addrs=127.0.212.126:33693
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:11.639808 6944 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:11.641335 6944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:11.652611 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:55:12.294267 6940 heartbeater.cc:499] Master 127.0.212.126:33693 was elected leader, sending a full tablet report...
W20250809 07:55:11.653340 6951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:13.031163 6952 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1377 milliseconds
W20250809 07:55:13.032961 6953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:13.035179 6944 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.383s user 0.528s sys 0.841s
W20250809 07:55:13.035437 6944 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.383s user 0.528s sys 0.841s
I20250809 07:55:13.035689 6944 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:13.036991 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:13.040591 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:13.041990 6944 hybrid_clock.cc:648] HybridClock initialized: now 1754726113041904 us; error 73 us; skew 500 ppm
I20250809 07:55:13.043085 6944 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:13.053429 6944 webserver.cc:489] Webserver started at http://127.0.212.67:42273/ using document root <none> and password file <none>
I20250809 07:55:13.054697 6944 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:13.055033 6944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:13.055660 6944 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:55:13.062119 6944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/instance:
uuid: "8f56d4ccd8d045978981974d22829a1d"
format_stamp: "Formatted at 2025-08-09 07:55:13 on dist-test-slave-tkp6"
I20250809 07:55:13.063585 6944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal/instance:
uuid: "8f56d4ccd8d045978981974d22829a1d"
format_stamp: "Formatted at 2025-08-09 07:55:13 on dist-test-slave-tkp6"
I20250809 07:55:13.072548 6944 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250809 07:55:13.079855 6960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:13.080950 6944 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250809 07:55:13.081326 6944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
uuid: "8f56d4ccd8d045978981974d22829a1d"
format_stamp: "Formatted at 2025-08-09 07:55:13 on dist-test-slave-tkp6"
I20250809 07:55:13.081732 6944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:13.154160 6944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:13.155568 6944 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:13.156004 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:13.158524 6944 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:13.162601 6944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 07:55:13.162817 6944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:13.163060 6944 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 07:55:13.163216 6944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:13.288264 6944 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:38049
I20250809 07:55:13.288489 7072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:38049 every 8 connection(s)
I20250809 07:55:13.291775 6944 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb
I20250809 07:55:13.297910 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 6944
I20250809 07:55:13.298388 849 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal/instance
I20250809 07:55:13.311794 7073 heartbeater.cc:344] Connected to a master server at 127.0.212.126:33693
I20250809 07:55:13.312192 7073 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:13.313098 7073 heartbeater.cc:507] Master 127.0.212.126:33693 requested a full tablet report, sending...
I20250809 07:55:13.314977 6618 ts_manager.cc:194] Registered new tserver with Master: 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049)
I20250809 07:55:13.316327 6618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:33031
I20250809 07:55:13.318041 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:55:13.345494 849 test_util.cc:276] Using random seed: 44695492
I20250809 07:55:13.381300 6618 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51478:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250809 07:55:13.383904 6618 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 07:55:13.435958 7008 tablet_service.cc:1468] Processing CreateTablet for tablet a54b244415b542b989eb8b813ae16638 (DEFAULT_TABLE table=pre_rebuild [id=2afe7c63b8214833aa750619afbf7ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:13.436609 6741 tablet_service.cc:1468] Processing CreateTablet for tablet a54b244415b542b989eb8b813ae16638 (DEFAULT_TABLE table=pre_rebuild [id=2afe7c63b8214833aa750619afbf7ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:13.438127 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54b244415b542b989eb8b813ae16638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:13.438418 6741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54b244415b542b989eb8b813ae16638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:13.444079 6875 tablet_service.cc:1468] Processing CreateTablet for tablet a54b244415b542b989eb8b813ae16638 (DEFAULT_TABLE table=pre_rebuild [id=2afe7c63b8214833aa750619afbf7ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:13.445850 6875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54b244415b542b989eb8b813ae16638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:13.464565 7098 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Bootstrap starting.
I20250809 07:55:13.463804 7097 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Bootstrap starting.
I20250809 07:55:13.470113 7098 tablet_bootstrap.cc:654] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:13.472062 7097 tablet_bootstrap.cc:654] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:13.472376 7098 log.cc:826] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:13.472119 7099 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Bootstrap starting.
I20250809 07:55:13.474316 7097 log.cc:826] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:13.477856 7098 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: No bootstrap required, opened a new log
I20250809 07:55:13.478744 7098 ts_tablet_manager.cc:1397] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20250809 07:55:13.479501 7097 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: No bootstrap required, opened a new log
I20250809 07:55:13.480000 7097 ts_tablet_manager.cc:1397] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20250809 07:55:13.480963 7099 tablet_bootstrap.cc:654] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:13.483124 7099 log.cc:826] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:13.488417 7099 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: No bootstrap required, opened a new log
I20250809 07:55:13.488904 7099 ts_tablet_manager.cc:1397] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.002s
I20250809 07:55:13.499953 7098 raft_consensus.cc:357] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.500767 7098 raft_consensus.cc:383] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:13.501012 7098 raft_consensus.cc:738] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abe071f97cf4aa6a5a4d2ac8e18eff2, State: Initialized, Role: FOLLOWER
I20250809 07:55:13.501719 7098 consensus_queue.cc:260] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.504802 7097 raft_consensus.cc:357] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.505533 7098 ts_tablet_manager.cc:1428] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Time spent starting tablet: real 0.027s user 0.019s sys 0.004s
I20250809 07:55:13.505684 7097 raft_consensus.cc:383] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:13.506007 7097 raft_consensus.cc:738] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56d4ccd8d045978981974d22829a1d, State: Initialized, Role: FOLLOWER
I20250809 07:55:13.506791 7097 consensus_queue.cc:260] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.510658 7073 heartbeater.cc:499] Master 127.0.212.126:33693 was elected leader, sending a full tablet report...
I20250809 07:55:13.512004 7097 ts_tablet_manager.cc:1428] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Time spent starting tablet: real 0.032s user 0.029s sys 0.002s
I20250809 07:55:13.514042 7099 raft_consensus.cc:357] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.514691 7099 raft_consensus.cc:383] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:13.514917 7099 raft_consensus.cc:738] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4e315577314ed8a6638ed0160b2896, State: Initialized, Role: FOLLOWER
I20250809 07:55:13.515602 7099 consensus_queue.cc:260] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.519174 7099 ts_tablet_manager.cc:1428] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Time spent starting tablet: real 0.030s user 0.028s sys 0.002s
W20250809 07:55:13.522321 6941 tablet.cc:2378] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 07:55:13.545696 7074 tablet.cc:2378] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:55:13.680577 7104 raft_consensus.cc:491] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:55:13.681066 7104 raft_consensus.cc:513] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.683447 7104 leader_election.cc:290] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865), 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
W20250809 07:55:13.688596 6807 tablet.cc:2378] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:55:13.692273 6761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" is_pre_election: true
I20250809 07:55:13.692977 6761 raft_consensus.cc:2466] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 0.
I20250809 07:55:13.693851 6895 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896" is_pre_election: true
I20250809 07:55:13.694181 6964 leader_election.cc:304] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abe071f97cf4aa6a5a4d2ac8e18eff2, 8f56d4ccd8d045978981974d22829a1d; no voters:
I20250809 07:55:13.694684 6895 raft_consensus.cc:2466] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 0.
I20250809 07:55:13.694953 7104 raft_consensus.cc:2802] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:55:13.695194 7104 raft_consensus.cc:491] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:55:13.695430 7104 raft_consensus.cc:3058] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:13.699595 7104 raft_consensus.cc:513] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.700969 7104 leader_election.cc:290] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 election: Requested vote from peers 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865), 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
I20250809 07:55:13.701565 6761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2"
I20250809 07:55:13.701843 6895 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896"
I20250809 07:55:13.702041 6761 raft_consensus.cc:3058] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:13.702353 6895 raft_consensus.cc:3058] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:13.708407 6761 raft_consensus.cc:2466] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 1.
I20250809 07:55:13.708483 6895 raft_consensus.cc:2466] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 1.
I20250809 07:55:13.709316 6964 leader_election.cc:304] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abe071f97cf4aa6a5a4d2ac8e18eff2, 8f56d4ccd8d045978981974d22829a1d; no voters:
I20250809 07:55:13.709942 7104 raft_consensus.cc:2802] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:55:13.711310 7104 raft_consensus.cc:695] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 LEADER]: Becoming Leader. State: Replica: 8f56d4ccd8d045978981974d22829a1d, State: Running, Role: LEADER
I20250809 07:55:13.712006 7104 consensus_queue.cc:237] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:13.720497 6618 catalog_manager.cc:5582] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f56d4ccd8d045978981974d22829a1d (127.0.212.67). New cstate: current_term: 1 leader_uuid: "8f56d4ccd8d045978981974d22829a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } health_report { overall_health: UNKNOWN } } }
I20250809 07:55:13.872016 6895 raft_consensus.cc:1273] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Refusing update from remote peer 8f56d4ccd8d045978981974d22829a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 07:55:13.872403 6761 raft_consensus.cc:1273] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Refusing update from remote peer 8f56d4ccd8d045978981974d22829a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 07:55:13.873741 7107 consensus_queue.cc:1035] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:55:13.874322 7104 consensus_queue.cc:1035] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:55:13.900511 7114 mvcc.cc:204] Tried to move back new op lower bound from 7187358162404405248 to 7187358161770713088. Current Snapshot: MvccSnapshot[applied={T|T < 7187358162404405248}]
I20250809 07:55:18.703442 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6585
W20250809 07:55:18.970121 6806 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:33693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:33693: connect: Connection refused (error 111)
W20250809 07:55:18.973572 6940 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:33693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:33693: connect: Connection refused (error 111)
W20250809 07:55:18.977840 7073 heartbeater.cc:646] Failed to heartbeat to 127.0.212.126:33693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.212.126:33693: connect: Connection refused (error 111)
W20250809 07:55:19.026510 7146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:19.027112 7146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:19.057967 7146 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 07:55:20.490777 7146 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.396s user 0.526s sys 0.865s
W20250809 07:55:20.491070 7146 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.396s user 0.526s sys 0.865s
I20250809 07:55:20.570513 7146 minidump.cc:252] Setting minidump size limit to 20M
I20250809 07:55:20.572878 7146 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:20.573936 7146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:20.584262 7180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:20.586145 7181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:20.720669 7183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:55:20.720753 7146 server_base.cc:1047] running on GCE node
I20250809 07:55:20.721859 7146 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250809 07:55:20.722297 7146 hybrid_clock.cc:648] HybridClock initialized: now 1754726120722275 us; error 127016 us; skew 500 ppm
I20250809 07:55:20.722947 7146 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:20.726913 7146 webserver.cc:489] Webserver started at http://0.0.0.0:43589/ using document root <none> and password file <none>
I20250809 07:55:20.727705 7146 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:20.727967 7146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:20.728375 7146 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 07:55:20.732547 7146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/instance:
uuid: "449de2e0e5eb464e8b0af2fc373b50a5"
format_stamp: "Formatted at 2025-08-09 07:55:20 on dist-test-slave-tkp6"
I20250809 07:55:20.733579 7146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal/instance:
uuid: "449de2e0e5eb464e8b0af2fc373b50a5"
format_stamp: "Formatted at 2025-08-09 07:55:20 on dist-test-slave-tkp6"
I20250809 07:55:20.739499 7146 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250809 07:55:20.744374 7189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:20.745187 7146 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250809 07:55:20.745460 7146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
uuid: "449de2e0e5eb464e8b0af2fc373b50a5"
format_stamp: "Formatted at 2025-08-09 07:55:20 on dist-test-slave-tkp6"
I20250809 07:55:20.745725 7146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:20.850610 7146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:20.851981 7146 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:20.852375 7146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:20.856652 7146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:20.869971 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Bootstrap starting.
I20250809 07:55:20.874552 7146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:20.876149 7146 log.cc:826] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:20.880120 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: No bootstrap required, opened a new log
I20250809 07:55:20.895390 7146 raft_consensus.cc:357] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER }
I20250809 07:55:20.895893 7146 raft_consensus.cc:383] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:20.896085 7146 raft_consensus.cc:738] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449de2e0e5eb464e8b0af2fc373b50a5, State: Initialized, Role: FOLLOWER
I20250809 07:55:20.896665 7146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER }
I20250809 07:55:20.897094 7146 raft_consensus.cc:397] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:55:20.897298 7146 raft_consensus.cc:491] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:55:20.897544 7146 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:20.901557 7146 raft_consensus.cc:513] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER }
I20250809 07:55:20.902223 7146 leader_election.cc:304] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449de2e0e5eb464e8b0af2fc373b50a5; no voters:
I20250809 07:55:20.903816 7146 leader_election.cc:290] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 07:55:20.904035 7196 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:55:20.906949 7196 raft_consensus.cc:695] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 LEADER]: Becoming Leader. State: Replica: 449de2e0e5eb464e8b0af2fc373b50a5, State: Running, Role: LEADER
I20250809 07:55:20.907709 7196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER }
I20250809 07:55:20.914718 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449de2e0e5eb464e8b0af2fc373b50a5. Latest consensus state: current_term: 1 leader_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER } }
I20250809 07:55:20.915187 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:20.916217 7197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER } }
I20250809 07:55:20.916720 7197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:20.926374 7146 tablet_replica.cc:331] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: stopping tablet replica
I20250809 07:55:20.926860 7146 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 LEADER]: Raft consensus shutting down.
I20250809 07:55:20.927273 7146 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250809 07:55:20.929142 7146 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250809 07:55:20.929569 7146 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250809 07:55:20.954093 7146 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250809 07:55:21.981055 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6677
I20250809 07:55:22.009055 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6810
I20250809 07:55:22.036474 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 6944
I20250809 07:55:22.070341 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:33693
--webserver_interface=127.0.212.126
--webserver_port=39305
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.212.126:33693 with env {}
W20250809 07:55:22.357362 7208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:22.357910 7208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:22.358358 7208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:22.388515 7208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 07:55:22.388818 7208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:22.389066 7208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 07:55:22.389300 7208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 07:55:22.423872 7208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.212.126:33693
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.212.126:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.212.126
--webserver_port=39305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:22.425123 7208 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:22.426656 7208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:22.436553 7214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:22.437315 7215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:23.544770 7217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:23.546787 7216 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1105 milliseconds
I20250809 07:55:23.546943 7208 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:23.548137 7208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:23.550532 7208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:23.551833 7208 hybrid_clock.cc:648] HybridClock initialized: now 1754726123551804 us; error 38 us; skew 500 ppm
I20250809 07:55:23.552597 7208 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:23.558266 7208 webserver.cc:489] Webserver started at http://127.0.212.126:39305/ using document root <none> and password file <none>
I20250809 07:55:23.559125 7208 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:23.559345 7208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:23.566606 7208 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250809 07:55:23.571326 7224 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:23.572605 7208 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 07:55:23.572983 7208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
uuid: "449de2e0e5eb464e8b0af2fc373b50a5"
format_stamp: "Formatted at 2025-08-09 07:55:20 on dist-test-slave-tkp6"
I20250809 07:55:23.575419 7208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:23.623952 7208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:23.625342 7208 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:23.625746 7208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:23.690624 7208 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.126:33693
I20250809 07:55:23.690689 7275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.126:33693 every 8 connection(s)
I20250809 07:55:23.693305 7208 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb
I20250809 07:55:23.700585 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 7208
I20250809 07:55:23.702479 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.65:34865
--local_ip_for_outbound_sockets=127.0.212.65
--tserver_master_addrs=127.0.212.126:33693
--webserver_port=44659
--webserver_interface=127.0.212.65
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 07:55:23.703486 7276 sys_catalog.cc:263] Verifying existing consensus state
I20250809 07:55:23.717535 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Bootstrap starting.
I20250809 07:55:23.727066 7276 log.cc:826] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:23.738306 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:55:23.738977 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Bootstrap complete.
I20250809 07:55:23.758685 7276 raft_consensus.cc:357] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:23.759442 7276 raft_consensus.cc:738] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449de2e0e5eb464e8b0af2fc373b50a5, State: Initialized, Role: FOLLOWER
I20250809 07:55:23.760322 7276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:23.760778 7276 raft_consensus.cc:397] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 07:55:23.761027 7276 raft_consensus.cc:491] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 07:55:23.761324 7276 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:55:23.765169 7276 raft_consensus.cc:513] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:23.765801 7276 leader_election.cc:304] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449de2e0e5eb464e8b0af2fc373b50a5; no voters:
I20250809 07:55:23.767750 7276 leader_election.cc:290] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 07:55:23.768265 7280 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:55:23.771111 7280 raft_consensus.cc:695] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [term 2 LEADER]: Becoming Leader. State: Replica: 449de2e0e5eb464e8b0af2fc373b50a5, State: Running, Role: LEADER
I20250809 07:55:23.771973 7280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } }
I20250809 07:55:23.772584 7276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 07:55:23.781517 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449de2e0e5eb464e8b0af2fc373b50a5. Latest consensus state: current_term: 2 leader_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } } }
I20250809 07:55:23.782773 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449de2e0e5eb464e8b0af2fc373b50a5" member_type: VOTER last_known_addr { host: "127.0.212.126" port: 33693 } } }
I20250809 07:55:23.783459 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:23.785847 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5 [sys.catalog]: This master's current role is: LEADER
I20250809 07:55:23.791308 7286 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 07:55:23.802094 7286 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=9d4599c974f741eb89585d7442f52499]
I20250809 07:55:23.809158 7286 tablet_loader.cc:96] loaded metadata for tablet a54b244415b542b989eb8b813ae16638 (table pre_rebuild [id=9d4599c974f741eb89585d7442f52499])
I20250809 07:55:23.811280 7286 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 07:55:23.837325 7286 catalog_manager.cc:1349] Generated new cluster ID: 15347feca0e84f19acc65c20e1e0f1c2
I20250809 07:55:23.837630 7286 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 07:55:23.859212 7286 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 07:55:23.861059 7286 catalog_manager.cc:1506] Loading token signing keys...
I20250809 07:55:23.889932 7286 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Generated new TSK 0
I20250809 07:55:23.890946 7286 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250809 07:55:24.031216 7278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:24.031700 7278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:24.032241 7278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:24.062721 7278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:24.063552 7278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.65
I20250809 07:55:24.096905 7278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.65:34865
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.212.65
--webserver_port=44659
--tserver_master_addrs=127.0.212.126:33693
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.65
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:24.098132 7278 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:24.099663 7278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:24.111930 7304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:25.513983 7303 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7278
W20250809 07:55:25.525432 7303 kernel_stack_watchdog.cc:198] Thread 7278 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 07:55:24.112674 7305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:25.527616 7278 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.415s user 0.438s sys 0.950s
W20250809 07:55:25.528113 7278 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.416s user 0.438s sys 0.951s
W20250809 07:55:25.528285 7307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:25.533119 7306 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1418 milliseconds
I20250809 07:55:25.533146 7278 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 07:55:25.534366 7278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:25.536422 7278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:25.537755 7278 hybrid_clock.cc:648] HybridClock initialized: now 1754726125537724 us; error 39 us; skew 500 ppm
I20250809 07:55:25.538496 7278 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:25.544436 7278 webserver.cc:489] Webserver started at http://127.0.212.65:44659/ using document root <none> and password file <none>
I20250809 07:55:25.545323 7278 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:25.545542 7278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:25.552969 7278 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250809 07:55:25.557513 7314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:25.558465 7278 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250809 07:55:25.558755 7278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2"
format_stamp: "Formatted at 2025-08-09 07:55:09 on dist-test-slave-tkp6"
I20250809 07:55:25.560607 7278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:25.620811 7278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:25.622177 7278 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:25.622591 7278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:25.625120 7278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:25.631273 7321 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 07:55:25.638190 7278 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 07:55:25.638392 7278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250809 07:55:25.638675 7278 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 07:55:25.642948 7278 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 07:55:25.643179 7278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250809 07:55:25.643545 7321 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Bootstrap starting.
I20250809 07:55:25.816326 7278 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.65:34865
I20250809 07:55:25.816475 7427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.65:34865 every 8 connection(s)
I20250809 07:55:25.819954 7278 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb
I20250809 07:55:25.827389 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 7278
I20250809 07:55:25.829463 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.66:46189
--local_ip_for_outbound_sockets=127.0.212.66
--tserver_master_addrs=127.0.212.126:33693
--webserver_port=35393
--webserver_interface=127.0.212.66
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 07:55:25.879772 7428 heartbeater.cc:344] Connected to a master server at 127.0.212.126:33693
I20250809 07:55:25.880254 7428 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:25.881372 7428 heartbeater.cc:507] Master 127.0.212.126:33693 requested a full tablet report, sending...
I20250809 07:55:25.885915 7241 ts_manager.cc:194] Registered new tserver with Master: 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865)
I20250809 07:55:25.893204 7241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.65:53741
I20250809 07:55:25.940625 7321 log.cc:826] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Log is configured to *not* fsync() on all Append() calls
W20250809 07:55:26.155495 7432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:26.156025 7432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:26.156522 7432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:26.187307 7432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:26.188172 7432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.66
I20250809 07:55:26.221982 7432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.66:46189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.212.66
--webserver_port=35393
--tserver_master_addrs=127.0.212.126:33693
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.66
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:26.223249 7432 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:26.224779 7432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:26.236485 7439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:55:26.897320 7428 heartbeater.cc:499] Master 127.0.212.126:33693 was elected leader, sending a full tablet report...
W20250809 07:55:27.637928 7438 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7432
W20250809 07:55:27.723606 7432 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.486s user 0.540s sys 0.936s
W20250809 07:55:26.238564 7440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:27.723987 7432 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.487s user 0.540s sys 0.936s
W20250809 07:55:27.731298 7443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:55:27.731652 7432 server_base.cc:1047] running on GCE node
I20250809 07:55:27.732697 7432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:27.734683 7432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:27.736011 7432 hybrid_clock.cc:648] HybridClock initialized: now 1754726127735977 us; error 33 us; skew 500 ppm
I20250809 07:55:27.736810 7432 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:27.742396 7432 webserver.cc:489] Webserver started at http://127.0.212.66:35393/ using document root <none> and password file <none>
I20250809 07:55:27.743281 7432 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:27.743498 7432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:27.751057 7432 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20250809 07:55:27.755553 7449 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:27.756628 7432 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250809 07:55:27.756948 7432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
uuid: "0a4e315577314ed8a6638ed0160b2896"
format_stamp: "Formatted at 2025-08-09 07:55:11 on dist-test-slave-tkp6"
I20250809 07:55:27.758754 7432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:27.806039 7432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:27.807405 7432 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:27.807885 7432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:27.810321 7432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:27.816428 7456 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 07:55:27.823704 7432 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 07:55:27.823966 7432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250809 07:55:27.824239 7432 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 07:55:27.828576 7432 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 07:55:27.828866 7432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250809 07:55:27.829170 7456 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Bootstrap starting.
I20250809 07:55:28.003569 7432 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.66:46189
I20250809 07:55:28.003870 7562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.66:46189 every 8 connection(s)
I20250809 07:55:28.007359 7432 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb
I20250809 07:55:28.011426 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 7432
I20250809 07:55:28.013494 849 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8reSeH/build/tsan/bin/kudu
/tmp/dist-test-task8reSeH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.212.67:38049
--local_ip_for_outbound_sockets=127.0.212.67
--tserver_master_addrs=127.0.212.126:33693
--webserver_port=42273
--webserver_interface=127.0.212.67
--builtin_ntp_servers=127.0.212.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 07:55:28.047626 7321 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:55:28.048705 7321 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Bootstrap complete.
I20250809 07:55:28.050366 7321 ts_tablet_manager.cc:1397] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Time spent bootstrapping tablet: real 2.407s user 2.346s sys 0.056s
I20250809 07:55:28.066143 7321 raft_consensus.cc:357] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:28.069056 7321 raft_consensus.cc:738] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abe071f97cf4aa6a5a4d2ac8e18eff2, State: Initialized, Role: FOLLOWER
I20250809 07:55:28.070075 7321 consensus_queue.cc:260] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:28.081794 7321 ts_tablet_manager.cc:1428] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Time spent starting tablet: real 0.031s user 0.020s sys 0.010s
I20250809 07:55:28.082166 7563 heartbeater.cc:344] Connected to a master server at 127.0.212.126:33693
I20250809 07:55:28.082624 7563 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:28.083806 7563 heartbeater.cc:507] Master 127.0.212.126:33693 requested a full tablet report, sending...
I20250809 07:55:28.087424 7241 ts_manager.cc:194] Registered new tserver with Master: 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
I20250809 07:55:28.090279 7241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.66:50073
I20250809 07:55:28.115777 7456 log.cc:826] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Log is configured to *not* fsync() on all Append() calls
W20250809 07:55:28.353652 7565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 07:55:28.354120 7565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 07:55:28.354586 7565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 07:55:28.385196 7565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 07:55:28.386075 7565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.212.67
I20250809 07:55:28.420135 7565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.212.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.212.67:38049
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.212.67
--webserver_port=42273
--tserver_master_addrs=127.0.212.126:33693
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.212.67
--log_dir=/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 07:43:24 UTC on 5fd53c4cbb9d
build id 7481
TSAN enabled
I20250809 07:55:28.421334 7565 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 07:55:28.422829 7565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 07:55:28.434240 7575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 07:55:29.094308 7563 heartbeater.cc:499] Master 127.0.212.126:33693 was elected leader, sending a full tablet report...
I20250809 07:55:29.506578 7581 raft_consensus.cc:491] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:55:29.507513 7581 raft_consensus.cc:513] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:29.528302 7581 leader_election.cc:290] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049), 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
W20250809 07:55:29.553860 7316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.212.67:38049: connect: Connection refused (error 111)
I20250809 07:55:29.582837 7518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896" is_pre_election: true
W20250809 07:55:29.598351 7316 leader_election.cc:336] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049): Network error: Client connection negotiation failed: client connection to 127.0.212.67:38049: connect: Connection refused (error 111)
W20250809 07:55:29.601882 7316 leader_election.cc:343] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189): Illegal state: must be running to vote when last-logged opid is not known
I20250809 07:55:29.602746 7316 leader_election.cc:304] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7abe071f97cf4aa6a5a4d2ac8e18eff2; no voters: 0a4e315577314ed8a6638ed0160b2896, 8f56d4ccd8d045978981974d22829a1d
I20250809 07:55:29.604835 7581 raft_consensus.cc:2747] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250809 07:55:28.435359 7576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:29.835796 7574 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7565
W20250809 07:55:30.036216 7578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 07:55:30.036314 7565 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.600s user 0.548s sys 0.838s
W20250809 07:55:30.038262 7565 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.603s user 0.548s sys 0.839s
I20250809 07:55:30.038554 7565 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250809 07:55:30.036688 7577 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1599 milliseconds
I20250809 07:55:30.042519 7565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 07:55:30.045078 7565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 07:55:30.046559 7565 hybrid_clock.cc:648] HybridClock initialized: now 1754726130046516 us; error 43 us; skew 500 ppm
I20250809 07:55:30.047588 7565 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 07:55:30.054630 7565 webserver.cc:489] Webserver started at http://127.0.212.67:42273/ using document root <none> and password file <none>
I20250809 07:55:30.055512 7565 fs_manager.cc:362] Metadata directory not provided
I20250809 07:55:30.055757 7565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 07:55:30.063295 7565 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250809 07:55:30.067844 7590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 07:55:30.068819 7565 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250809 07:55:30.069100 7565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data,/tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
uuid: "8f56d4ccd8d045978981974d22829a1d"
format_stamp: "Formatted at 2025-08-09 07:55:13 on dist-test-slave-tkp6"
I20250809 07:55:30.070940 7565 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 07:55:30.131999 7565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 07:55:30.133394 7565 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 07:55:30.133800 7565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 07:55:30.136302 7565 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 07:55:30.142256 7597 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 07:55:30.149287 7565 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 07:55:30.149477 7565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250809 07:55:30.149752 7565 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 07:55:30.154069 7565 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 07:55:30.154309 7565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250809 07:55:30.154670 7597 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Bootstrap starting.
I20250809 07:55:30.325846 7565 rpc_server.cc:307] RPC server started. Bound to: 127.0.212.67:38049
I20250809 07:55:30.325987 7703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.212.67:38049 every 8 connection(s)
I20250809 07:55:30.329471 7565 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb
I20250809 07:55:30.337268 849 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8reSeH/build/tsan/bin/kudu as pid 7565
I20250809 07:55:30.420912 7704 heartbeater.cc:344] Connected to a master server at 127.0.212.126:33693
I20250809 07:55:30.421403 7704 heartbeater.cc:461] Registering TS with master...
I20250809 07:55:30.422586 7704 heartbeater.cc:507] Master 127.0.212.126:33693 requested a full tablet report, sending...
I20250809 07:55:30.426276 7241 ts_manager.cc:194] Registered new tserver with Master: 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049)
I20250809 07:55:30.429689 7241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.212.67:60567
I20250809 07:55:30.433622 849 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 07:55:30.451884 7597 log.cc:826] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Log is configured to *not* fsync() on all Append() calls
I20250809 07:55:30.677182 7456 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:55:30.677907 7456 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Bootstrap complete.
I20250809 07:55:30.678978 7456 ts_tablet_manager.cc:1397] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Time spent bootstrapping tablet: real 2.850s user 2.731s sys 0.092s
I20250809 07:55:30.684587 7456 raft_consensus.cc:357] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:30.686435 7456 raft_consensus.cc:738] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4e315577314ed8a6638ed0160b2896, State: Initialized, Role: FOLLOWER
I20250809 07:55:30.687111 7456 consensus_queue.cc:260] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:30.690305 7456 ts_tablet_manager.cc:1428] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20250809 07:55:31.434592 7704 heartbeater.cc:499] Master 127.0.212.126:33693 was elected leader, sending a full tablet report...
I20250809 07:55:31.471853 7718 raft_consensus.cc:491] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:55:31.472246 7718 raft_consensus.cc:513] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:31.473673 7718 leader_election.cc:290] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049), 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
I20250809 07:55:31.476568 7518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896" is_pre_election: true
I20250809 07:55:31.477290 7518 raft_consensus.cc:2466] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abe071f97cf4aa6a5a4d2ac8e18eff2 in term 1.
I20250809 07:55:31.478344 7316 leader_election.cc:304] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4e315577314ed8a6638ed0160b2896, 7abe071f97cf4aa6a5a4d2ac8e18eff2; no voters:
I20250809 07:55:31.479104 7718 raft_consensus.cc:2802] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250809 07:55:31.479413 7718 raft_consensus.cc:491] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:55:31.479712 7718 raft_consensus.cc:3058] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:55:31.488194 7718 raft_consensus.cc:513] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:31.489908 7718 leader_election.cc:290] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 election: Requested vote from peers 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049), 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189)
I20250809 07:55:31.491001 7518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896"
I20250809 07:55:31.491554 7518 raft_consensus.cc:3058] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Advancing to term 2
I20250809 07:55:31.497534 7518 raft_consensus.cc:2466] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abe071f97cf4aa6a5a4d2ac8e18eff2 in term 2.
I20250809 07:55:31.498359 7316 leader_election.cc:304] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4e315577314ed8a6638ed0160b2896, 7abe071f97cf4aa6a5a4d2ac8e18eff2; no voters:
I20250809 07:55:31.498939 7718 raft_consensus.cc:2802] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 07:55:31.489264 7659 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "8f56d4ccd8d045978981974d22829a1d" is_pre_election: true
I20250809 07:55:31.491425 7658 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54b244415b542b989eb8b813ae16638" candidate_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "8f56d4ccd8d045978981974d22829a1d"
I20250809 07:55:31.500566 7718 raft_consensus.cc:695] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 LEADER]: Becoming Leader. State: Replica: 7abe071f97cf4aa6a5a4d2ac8e18eff2, State: Running, Role: LEADER
W20250809 07:55:31.501441 7316 leader_election.cc:343] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049): Illegal state: must be running to vote when last-logged opid is not known
I20250809 07:55:31.501469 7718 consensus_queue.cc:237] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
W20250809 07:55:31.502748 7316 leader_election.cc:343] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049): Illegal state: must be running to vote when last-logged opid is not known
I20250809 07:55:31.513120 7241 catalog_manager.cc:5582] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 reported cstate change: term changed from 0 to 2, leader changed from <none> to 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65), VOTER 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66) added, VOTER 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65) added, VOTER 8f56d4ccd8d045978981974d22829a1d (127.0.212.67) added. New cstate: current_term: 2 leader_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } health_report { overall_health: UNKNOWN } } }
W20250809 07:55:31.970575 7316 consensus_peers.cc:489] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 -> Peer 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049): Couldn't send request to peer 8f56d4ccd8d045978981974d22829a1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250809 07:55:32.024389 7518 raft_consensus.cc:1273] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 2 FOLLOWER]: Refusing update from remote peer 7abe071f97cf4aa6a5a4d2ac8e18eff2: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250809 07:55:32.026206 7718 consensus_queue.cc:1035] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250809 07:55:32.095289 7383 consensus_queue.cc:237] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:32.100841 7517 raft_consensus.cc:1273] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 2 FOLLOWER]: Refusing update from remote peer 7abe071f97cf4aa6a5a4d2ac8e18eff2: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250809 07:55:32.102315 7718 consensus_queue.cc:1035] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250809 07:55:32.109678 7718 raft_consensus.cc:2953] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 8f56d4ccd8d045978981974d22829a1d (127.0.212.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } }
I20250809 07:55:32.110062 7518 raft_consensus.cc:2953] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 8f56d4ccd8d045978981974d22829a1d (127.0.212.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } }
I20250809 07:55:32.120038 7228 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a54b244415b542b989eb8b813ae16638 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 07:55:32.121500 7241 catalog_manager.cc:5582] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 reported cstate change: config changed from index -1 to 207, VOTER 8f56d4ccd8d045978981974d22829a1d (127.0.212.67) evicted. New cstate: current_term: 2 leader_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } }
I20250809 07:55:32.184839 7383 consensus_queue.cc:237] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:32.190210 7718 raft_consensus.cc:2953] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } }
I20250809 07:55:32.207273 7228 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a54b244415b542b989eb8b813ae16638 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 07:55:32.212035 7240 catalog_manager.cc:5582] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 reported cstate change: config changed from index 207 to 208, VOTER 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66) evicted. New cstate: current_term: 2 leader_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } health_report { overall_health: HEALTHY } } }
I20250809 07:55:32.278278 7639 tablet_service.cc:1515] Processing DeleteTablet for tablet a54b244415b542b989eb8b813ae16638 with delete_type TABLET_DATA_TOMBSTONED (TS 8f56d4ccd8d045978981974d22829a1d not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:35778
I20250809 07:55:32.287084 7498 tablet_service.cc:1515] Processing DeleteTablet for tablet a54b244415b542b989eb8b813ae16638 with delete_type TABLET_DATA_TOMBSTONED (TS 0a4e315577314ed8a6638ed0160b2896 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:42714
W20250809 07:55:32.292410 7226 catalog_manager.cc:4908] TS 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049): delete failed for tablet a54b244415b542b989eb8b813ae16638 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a54b244415b542b989eb8b813ae16638 already in progress: opening tablet
I20250809 07:55:32.298038 7744 tablet_replica.cc:331] stopping tablet replica
I20250809 07:55:32.298985 7744 raft_consensus.cc:2241] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250809 07:55:32.299707 7744 raft_consensus.cc:2270] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250809 07:55:32.325568 7744 ts_tablet_manager.cc:1905] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 07:55:32.340471 7744 ts_tablet_manager.cc:1918] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250809 07:55:32.341002 7744 log.cc:1199] T a54b244415b542b989eb8b813ae16638 P 0a4e315577314ed8a6638ed0160b2896: Deleting WAL directory at /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/wal/wals/a54b244415b542b989eb8b813ae16638
W20250809 07:55:32.341611 849 scanner-internal.cc:458] Time spent opening tablet: real 1.875s user 0.006s sys 0.002s
I20250809 07:55:32.342893 7226 catalog_manager.cc:4928] TS 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189): tablet a54b244415b542b989eb8b813ae16638 (table pre_rebuild [id=9d4599c974f741eb89585d7442f52499]) successfully deleted
I20250809 07:55:32.543661 7597 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 07:55:32.544657 7597 tablet_bootstrap.cc:492] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Bootstrap complete.
I20250809 07:55:32.546329 7597 ts_tablet_manager.cc:1397] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Time spent bootstrapping tablet: real 2.392s user 2.290s sys 0.064s
I20250809 07:55:32.553911 7597 raft_consensus.cc:357] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:32.556859 7597 raft_consensus.cc:738] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56d4ccd8d045978981974d22829a1d, State: Initialized, Role: FOLLOWER
I20250809 07:55:32.557686 7597 consensus_queue.cc:260] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } } peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } }
I20250809 07:55:32.560854 7597 ts_tablet_manager.cc:1428] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20250809 07:55:32.565879 7639 tablet_service.cc:1515] Processing DeleteTablet for tablet a54b244415b542b989eb8b813ae16638 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:35778
I20250809 07:55:32.566872 7743 tablet_replica.cc:331] stopping tablet replica
I20250809 07:55:32.567575 7743 raft_consensus.cc:2241] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250809 07:55:32.568084 7743 raft_consensus.cc:2270] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250809 07:55:32.592989 7743 ts_tablet_manager.cc:1905] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 07:55:32.610702 7743 ts_tablet_manager.cc:1918] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250809 07:55:32.611091 7743 log.cc:1199] T a54b244415b542b989eb8b813ae16638 P 8f56d4ccd8d045978981974d22829a1d: Deleting WAL directory at /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/wal/wals/a54b244415b542b989eb8b813ae16638
I20250809 07:55:32.612782 7226 catalog_manager.cc:4928] TS 8f56d4ccd8d045978981974d22829a1d (127.0.212.67:38049): tablet a54b244415b542b989eb8b813ae16638 (table pre_rebuild [id=9d4599c974f741eb89585d7442f52499]) successfully deleted
I20250809 07:55:33.023461 7498 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:55:33.033097 7639 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:55:33.051167 7363 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
449de2e0e5eb464e8b0af2fc373b50a5 | 127.0.212.126:33693 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.212.84:44843 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0a4e315577314ed8a6638ed0160b2896 | 127.0.212.66:46189 | HEALTHY | <none> | 0 | 0
7abe071f97cf4aa6a5a4d2ac8e18eff2 | 127.0.212.65:34865 | HEALTHY | <none> | 1 | 0
8f56d4ccd8d045978981974d22829a1d | 127.0.212.67:38049 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.212.65 | experimental | 127.0.212.65:34865
local_ip_for_outbound_sockets | 127.0.212.66 | experimental | 127.0.212.66:46189
local_ip_for_outbound_sockets | 127.0.212.67 | experimental | 127.0.212.67:38049
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.212.65:34865
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.212.66:46189
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.212.67:38049
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.212.84:44843 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250809 07:55:33.317842 849 log_verifier.cc:126] Checking tablet a54b244415b542b989eb8b813ae16638
I20250809 07:55:33.591003 849 log_verifier.cc:177] Verified matching terms for 208 ops in tablet a54b244415b542b989eb8b813ae16638
I20250809 07:55:33.593400 7241 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38130:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250809 07:55:33.594074 7241 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38130:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250809 07:55:33.608693 7241 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 449de2e0e5eb464e8b0af2fc373b50a5: Sending DeleteTablet for 1 replicas of tablet a54b244415b542b989eb8b813ae16638
I20250809 07:55:33.610390 7363 tablet_service.cc:1515] Processing DeleteTablet for tablet a54b244415b542b989eb8b813ae16638 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-09 07:55:33 UTC) from {username='slave'} at 127.0.0.1:55826
I20250809 07:55:33.610594 849 test_util.cc:276] Using random seed: 64960565
I20250809 07:55:33.612183 7778 tablet_replica.cc:331] stopping tablet replica
I20250809 07:55:33.612960 7778 raft_consensus.cc:2241] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 LEADER]: Raft consensus shutting down.
I20250809 07:55:33.613619 7778 raft_consensus.cc:2270] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250809 07:55:33.637409 7241 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38168:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250809 07:55:33.639952 7241 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 07:55:33.644050 7778 ts_tablet_manager.cc:1905] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250809 07:55:33.659235 7778 ts_tablet_manager.cc:1918] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250809 07:55:33.659686 7778 log.cc:1199] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Deleting WAL directory at /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/wal/wals/a54b244415b542b989eb8b813ae16638
I20250809 07:55:33.660537 7778 ts_tablet_manager.cc:1939] T a54b244415b542b989eb8b813ae16638 P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Deleting consensus metadata
I20250809 07:55:33.662459 7498 tablet_service.cc:1468] Processing CreateTablet for tablet 65e5a716169245e5b3666a40c3b5d8de (DEFAULT_TABLE table=post_rebuild [id=6881ddd795d349539bb7c0c2d5898809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:33.663002 7639 tablet_service.cc:1468] Processing CreateTablet for tablet 65e5a716169245e5b3666a40c3b5d8de (DEFAULT_TABLE table=post_rebuild [id=6881ddd795d349539bb7c0c2d5898809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:33.663779 7498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e5a716169245e5b3666a40c3b5d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:33.663798 7228 catalog_manager.cc:4928] TS 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865): tablet a54b244415b542b989eb8b813ae16638 (table pre_rebuild [id=9d4599c974f741eb89585d7442f52499]) successfully deleted
I20250809 07:55:33.664402 7639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e5a716169245e5b3666a40c3b5d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:33.667546 7363 tablet_service.cc:1468] Processing CreateTablet for tablet 65e5a716169245e5b3666a40c3b5d8de (DEFAULT_TABLE table=post_rebuild [id=6881ddd795d349539bb7c0c2d5898809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 07:55:33.668789 7363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e5a716169245e5b3666a40c3b5d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 07:55:33.685601 7785 tablet_bootstrap.cc:492] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d: Bootstrap starting.
I20250809 07:55:33.689867 7786 tablet_bootstrap.cc:492] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Bootstrap starting.
I20250809 07:55:33.693917 7785 tablet_bootstrap.cc:654] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:33.695631 7787 tablet_bootstrap.cc:492] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896: Bootstrap starting.
I20250809 07:55:33.701658 7787 tablet_bootstrap.cc:654] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:33.705086 7786 tablet_bootstrap.cc:654] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Neither blocks nor log segments found. Creating new log.
I20250809 07:55:33.705235 7785 tablet_bootstrap.cc:492] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d: No bootstrap required, opened a new log
I20250809 07:55:33.705650 7785 ts_tablet_manager.cc:1397] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20250809 07:55:33.708415 7785 raft_consensus.cc:357] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.709266 7785 raft_consensus.cc:383] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:33.709602 7785 raft_consensus.cc:738] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56d4ccd8d045978981974d22829a1d, State: Initialized, Role: FOLLOWER
I20250809 07:55:33.710484 7785 consensus_queue.cc:260] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.716598 7787 tablet_bootstrap.cc:492] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896: No bootstrap required, opened a new log
I20250809 07:55:33.718601 7787 ts_tablet_manager.cc:1397] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20250809 07:55:33.721635 7787 raft_consensus.cc:357] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.723754 7786 tablet_bootstrap.cc:492] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2: No bootstrap required, opened a new log
I20250809 07:55:33.723945 7785 ts_tablet_manager.cc:1428] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d: Time spent starting tablet: real 0.018s user 0.012s sys 0.000s
I20250809 07:55:33.724233 7786 ts_tablet_manager.cc:1397] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.006s
I20250809 07:55:33.726791 7786 raft_consensus.cc:357] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.727602 7786 raft_consensus.cc:383] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:33.727955 7786 raft_consensus.cc:738] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abe071f97cf4aa6a5a4d2ac8e18eff2, State: Initialized, Role: FOLLOWER
I20250809 07:55:33.728741 7786 consensus_queue.cc:260] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.729878 7787 raft_consensus.cc:383] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 07:55:33.730268 7787 raft_consensus.cc:738] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4e315577314ed8a6638ed0160b2896, State: Initialized, Role: FOLLOWER
I20250809 07:55:33.730990 7787 consensus_queue.cc:260] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.745537 7786 ts_tablet_manager.cc:1428] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20250809 07:55:33.748083 7787 ts_tablet_manager.cc:1428] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896: Time spent starting tablet: real 0.029s user 0.008s sys 0.012s
W20250809 07:55:33.830368 7564 tablet.cc:2378] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 07:55:33.859576 7429 tablet.cc:2378] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:55:33.869959 7791 raft_consensus.cc:491] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 07:55:33.870503 7791 raft_consensus.cc:513] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.882258 7791 leader_election.cc:290] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189), 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865)
I20250809 07:55:33.896297 7383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5a716169245e5b3666a40c3b5d8de" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" is_pre_election: true
I20250809 07:55:33.896975 7383 raft_consensus.cc:2466] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 0.
I20250809 07:55:33.896950 7518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5a716169245e5b3666a40c3b5d8de" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896" is_pre_election: true
I20250809 07:55:33.897511 7518 raft_consensus.cc:2466] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 0.
I20250809 07:55:33.898186 7594 leader_election.cc:304] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abe071f97cf4aa6a5a4d2ac8e18eff2, 8f56d4ccd8d045978981974d22829a1d; no voters:
I20250809 07:55:33.898955 7791 raft_consensus.cc:2802] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 07:55:33.899304 7791 raft_consensus.cc:491] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 07:55:33.899624 7791 raft_consensus.cc:3058] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:33.906172 7791 raft_consensus.cc:513] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.908015 7791 leader_election.cc:290] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 election: Requested vote from peers 0a4e315577314ed8a6638ed0160b2896 (127.0.212.66:46189), 7abe071f97cf4aa6a5a4d2ac8e18eff2 (127.0.212.65:34865)
I20250809 07:55:33.908655 7518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5a716169245e5b3666a40c3b5d8de" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4e315577314ed8a6638ed0160b2896"
I20250809 07:55:33.909013 7383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5a716169245e5b3666a40c3b5d8de" candidate_uuid: "8f56d4ccd8d045978981974d22829a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2"
I20250809 07:55:33.909165 7518 raft_consensus.cc:3058] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:33.909503 7383 raft_consensus.cc:3058] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 0 FOLLOWER]: Advancing to term 1
I20250809 07:55:33.915225 7518 raft_consensus.cc:2466] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 1.
I20250809 07:55:33.915935 7383 raft_consensus.cc:2466] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56d4ccd8d045978981974d22829a1d in term 1.
I20250809 07:55:33.916468 7592 leader_election.cc:304] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4e315577314ed8a6638ed0160b2896, 8f56d4ccd8d045978981974d22829a1d; no voters:
W20250809 07:55:33.916725 7705 tablet.cc:2378] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 07:55:33.917312 7791 raft_consensus.cc:2802] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 1 FOLLOWER]: Leader election won for term 1
I20250809 07:55:33.919003 7791 raft_consensus.cc:695] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [term 1 LEADER]: Becoming Leader. State: Replica: 8f56d4ccd8d045978981974d22829a1d, State: Running, Role: LEADER
I20250809 07:55:33.919911 7791 consensus_queue.cc:237] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } }
I20250809 07:55:33.930915 7241 catalog_manager.cc:5582] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f56d4ccd8d045978981974d22829a1d (127.0.212.67). New cstate: current_term: 1 leader_uuid: "8f56d4ccd8d045978981974d22829a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56d4ccd8d045978981974d22829a1d" member_type: VOTER last_known_addr { host: "127.0.212.67" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 } health_report { overall_health: UNKNOWN } } }
I20250809 07:55:34.222179 7518 raft_consensus.cc:1273] T 65e5a716169245e5b3666a40c3b5d8de P 0a4e315577314ed8a6638ed0160b2896 [term 1 FOLLOWER]: Refusing update from remote peer 8f56d4ccd8d045978981974d22829a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 07:55:34.222924 7383 raft_consensus.cc:1273] T 65e5a716169245e5b3666a40c3b5d8de P 7abe071f97cf4aa6a5a4d2ac8e18eff2 [term 1 FOLLOWER]: Refusing update from remote peer 8f56d4ccd8d045978981974d22829a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 07:55:34.223948 7798 consensus_queue.cc:1035] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4e315577314ed8a6638ed0160b2896" member_type: VOTER last_known_addr { host: "127.0.212.66" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250809 07:55:34.225621 7798 consensus_queue.cc:1035] T 65e5a716169245e5b3666a40c3b5d8de P 8f56d4ccd8d045978981974d22829a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abe071f97cf4aa6a5a4d2ac8e18eff2" member_type: VOTER last_known_addr { host: "127.0.212.65" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 07:55:38.935215 7498 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:55:38.937196 7363 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 07:55:38.938020 7639 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
449de2e0e5eb464e8b0af2fc373b50a5 | 127.0.212.126:33693 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.212.84:44843 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0a4e315577314ed8a6638ed0160b2896 | 127.0.212.66:46189 | HEALTHY | <none> | 0 | 0
7abe071f97cf4aa6a5a4d2ac8e18eff2 | 127.0.212.65:34865 | HEALTHY | <none> | 0 | 0
8f56d4ccd8d045978981974d22829a1d | 127.0.212.67:38049 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.212.65 | experimental | 127.0.212.65:34865
local_ip_for_outbound_sockets | 127.0.212.66 | experimental | 127.0.212.66:46189
local_ip_for_outbound_sockets | 127.0.212.67 | experimental | 127.0.212.67:38049
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.212.65:34865
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.212.66:46189
server_dump_info_path | /tmp/dist-test-task8reSeH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754725960294695-849-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.212.67:38049
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.212.84:44843 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250809 07:55:39.148784 849 log_verifier.cc:126] Checking tablet 65e5a716169245e5b3666a40c3b5d8de
I20250809 07:55:39.924587 849 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 65e5a716169245e5b3666a40c3b5d8de
I20250809 07:55:39.925320 849 log_verifier.cc:126] Checking tablet a54b244415b542b989eb8b813ae16638
I20250809 07:55:39.925559 849 log_verifier.cc:177] Verified matching terms for 0 ops in tablet a54b244415b542b989eb8b813ae16638
I20250809 07:55:39.936441 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 7278
I20250809 07:55:39.972361 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 7432
I20250809 07:55:40.001212 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 7565
I20250809 07:55:40.034790 849 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8reSeH/build/tsan/bin/kudu with pid 7208
2025-08-09T07:55:40Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35046 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35046 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (179725 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST