Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250815 19:57:39.727216 20693 test_util.cc:276] Using random seed: -849638675
W20250815 19:57:40.920331 20693 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.150s user 0.442s sys 0.704s
W20250815 19:57:40.920679 20693 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.150s user 0.442s sys 0.704s
I20250815 19:57:40.922649 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 19:57:40.922804 20693 ts_itest-base.cc:116] --------------
I20250815 19:57:40.922966 20693 ts_itest-base.cc:117] 4 tablet servers
I20250815 19:57:40.923123 20693 ts_itest-base.cc:118] 3 replicas per TS
I20250815 19:57:40.923264 20693 ts_itest-base.cc:119] --------------
2025-08-15T19:57:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T19:57:40Z Disabled control of system clock
I20250815 19:57:40.962352 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:45645
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:45645 with env {}
W20250815 19:57:41.255824 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:41.256414 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:57:41.256918 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:41.287207 20707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:57:41.287494 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:57:41.287782 20707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:57:41.288021 20707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:57:41.321951 20707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:45645
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:57:41.323249 20707 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:57:41.324862 20707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:57:41.335351 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:41.335552 20714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:42.739589 20712 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 20707
W20250815 19:57:42.979251 20707 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.644s user 0.595s sys 1.042s
W20250815 19:57:42.980824 20707 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.646s user 0.596s sys 1.042s
W20250815 19:57:42.981514 20716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:42.984593 20715 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1644 milliseconds
I20250815 19:57:42.984666 20707 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:57:42.986120 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:57:42.988492 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:57:42.989809 20707 hybrid_clock.cc:648] HybridClock initialized: now 1755287862989791 us; error 46 us; skew 500 ppm
I20250815 19:57:42.990594 20707 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:57:42.996560 20707 webserver.cc:480] Webserver started at http://127.20.53.126:37419/ using document root <none> and password file <none>
I20250815 19:57:42.997465 20707 fs_manager.cc:362] Metadata directory not provided
I20250815 19:57:42.997664 20707 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:57:42.998101 20707 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:57:43.002281 20707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "37aa504c6787485d991e0b1d562f738f"
format_stamp: "Formatted at 2025-08-15 19:57:42 on dist-test-slave-d4bs"
I20250815 19:57:43.003300 20707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "37aa504c6787485d991e0b1d562f738f"
format_stamp: "Formatted at 2025-08-15 19:57:42 on dist-test-slave-d4bs"
I20250815 19:57:43.009867 20707 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250815 19:57:43.015172 20723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:43.016177 20707 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250815 19:57:43.016482 20707 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "37aa504c6787485d991e0b1d562f738f"
format_stamp: "Formatted at 2025-08-15 19:57:42 on dist-test-slave-d4bs"
I20250815 19:57:43.016899 20707 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:57:43.071410 20707 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:57:43.072865 20707 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:57:43.073295 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:57:43.142781 20707 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:45645
I20250815 19:57:43.142823 20774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:45645 every 8 connection(s)
I20250815 19:57:43.145546 20707 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:57:43.150125 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 20707
I20250815 19:57:43.150650 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 19:57:43.151042 20775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:57:43.171367 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Bootstrap starting.
I20250815 19:57:43.177268 20775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Neither blocks nor log segments found. Creating new log.
I20250815 19:57:43.179418 20775 log.cc:826] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Log is configured to *not* fsync() on all Append() calls
I20250815 19:57:43.184597 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: No bootstrap required, opened a new log
I20250815 19:57:43.203739 20775 raft_consensus.cc:357] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:43.204493 20775 raft_consensus.cc:383] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:57:43.204838 20775 raft_consensus.cc:738] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aa504c6787485d991e0b1d562f738f, State: Initialized, Role: FOLLOWER
I20250815 19:57:43.205634 20775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:43.206210 20775 raft_consensus.cc:397] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:57:43.206426 20775 raft_consensus.cc:491] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:57:43.206670 20775 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:57:43.210862 20775 raft_consensus.cc:513] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:43.211544 20775 leader_election.cc:304] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37aa504c6787485d991e0b1d562f738f; no voters:
I20250815 19:57:43.213127 20775 leader_election.cc:290] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:57:43.213861 20780 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:57:43.216020 20780 raft_consensus.cc:695] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 LEADER]: Becoming Leader. State: Replica: 37aa504c6787485d991e0b1d562f738f, State: Running, Role: LEADER
I20250815 19:57:43.217361 20780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:43.218191 20775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:57:43.228201 20782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37aa504c6787485d991e0b1d562f738f. Latest consensus state: current_term: 1 leader_uuid: "37aa504c6787485d991e0b1d562f738f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } } }
I20250815 19:57:43.228449 20781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37aa504c6787485d991e0b1d562f738f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } } }
I20250815 19:57:43.228940 20782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: This master's current role is: LEADER
I20250815 19:57:43.229292 20781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: This master's current role is: LEADER
I20250815 19:57:43.232491 20789 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:57:43.244762 20789 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:57:43.260979 20789 catalog_manager.cc:1349] Generated new cluster ID: d5b2005f5c0d4b81a05b71e2e6daa4bc
I20250815 19:57:43.261322 20789 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:57:43.295980 20789 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 19:57:43.297358 20789 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:57:43.309916 20789 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Generated new TSK 0
I20250815 19:57:43.310719 20789 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 19:57:43.325404 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--builtin_ntp_servers=127.20.53.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 19:57:43.651443 20800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:43.652163 20800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:57:43.653153 20800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:43.685271 20800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:57:43.686131 20800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:57:43.721131 20800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:57:43.722541 20800 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:57:43.724242 20800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:57:43.737255 20806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:43.738404 20807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:45.233170 20800 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.495s user 0.482s sys 0.979s
W20250815 19:57:45.233598 20800 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.495s user 0.482s sys 0.979s
W20250815 19:57:45.140201 20805 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 20800
W20250815 19:57:45.236032 20809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:45.239409 20808 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1499 milliseconds
I20250815 19:57:45.239444 20800 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:57:45.240962 20800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:57:45.243474 20800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:57:45.244894 20800 hybrid_clock.cc:648] HybridClock initialized: now 1755287865244840 us; error 52 us; skew 500 ppm
I20250815 19:57:45.245950 20800 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:57:45.253365 20800 webserver.cc:480] Webserver started at http://127.20.53.65:39851/ using document root <none> and password file <none>
I20250815 19:57:45.254623 20800 fs_manager.cc:362] Metadata directory not provided
I20250815 19:57:45.254902 20800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:57:45.255479 20800 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:57:45.262001 20800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b510897cd8674573b420c4490a85bb14"
format_stamp: "Formatted at 2025-08-15 19:57:45 on dist-test-slave-d4bs"
I20250815 19:57:45.263491 20800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b510897cd8674573b420c4490a85bb14"
format_stamp: "Formatted at 2025-08-15 19:57:45 on dist-test-slave-d4bs"
I20250815 19:57:45.272629 20800 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20250815 19:57:45.280160 20816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:45.281308 20800 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250815 19:57:45.281649 20800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b510897cd8674573b420c4490a85bb14"
format_stamp: "Formatted at 2025-08-15 19:57:45 on dist-test-slave-d4bs"
I20250815 19:57:45.281987 20800 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:57:45.335361 20800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:57:45.336966 20800 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:57:45.337404 20800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:57:45.340063 20800 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:57:45.345022 20800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:57:45.345244 20800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:45.345499 20800 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:57:45.345681 20800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:45.508189 20800 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:44477
I20250815 19:57:45.508371 20928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:44477 every 8 connection(s)
I20250815 19:57:45.510836 20800 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:57:45.516176 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 20800
I20250815 19:57:45.516661 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 19:57:45.524446 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--builtin_ntp_servers=127.20.53.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:57:45.541944 20929 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:45.542403 20929 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:45.543457 20929 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:45.545899 20740 ts_manager.cc:194] Registered new tserver with Master: b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
I20250815 19:57:45.547819 20740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:40855
W20250815 19:57:45.847998 20933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:45.848512 20933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:57:45.849043 20933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:45.880075 20933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:57:45.880955 20933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:57:45.915591 20933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:57:45.916922 20933 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:57:45.918534 20933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:57:45.930155 20939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:57:46.551079 20929 heartbeater.cc:499] Master 127.20.53.126:45645 was elected leader, sending a full tablet report...
W20250815 19:57:45.930743 20940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:47.259208 20942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:47.262149 20941 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1325 milliseconds
W20250815 19:57:47.262516 20933 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.332s user 0.442s sys 0.884s
W20250815 19:57:47.262867 20933 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.332s user 0.442s sys 0.884s
I20250815 19:57:47.263103 20933 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:57:47.264150 20933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:57:47.266445 20933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:57:47.267812 20933 hybrid_clock.cc:648] HybridClock initialized: now 1755287867267784 us; error 40 us; skew 500 ppm
I20250815 19:57:47.268596 20933 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:57:47.275729 20933 webserver.cc:480] Webserver started at http://127.20.53.66:41507/ using document root <none> and password file <none>
I20250815 19:57:47.276782 20933 fs_manager.cc:362] Metadata directory not provided
I20250815 19:57:47.277019 20933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:57:47.277451 20933 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:57:47.281865 20933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8815e8735d824256ae89701ac5baa20d"
format_stamp: "Formatted at 2025-08-15 19:57:47 on dist-test-slave-d4bs"
I20250815 19:57:47.282941 20933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8815e8735d824256ae89701ac5baa20d"
format_stamp: "Formatted at 2025-08-15 19:57:47 on dist-test-slave-d4bs"
I20250815 19:57:47.290884 20933 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250815 19:57:47.296785 20949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:47.297921 20933 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250815 19:57:47.298259 20933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8815e8735d824256ae89701ac5baa20d"
format_stamp: "Formatted at 2025-08-15 19:57:47 on dist-test-slave-d4bs"
I20250815 19:57:47.298596 20933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:57:47.362361 20933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:57:47.363811 20933 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:57:47.364241 20933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:57:47.366811 20933 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:57:47.370818 20933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:57:47.371042 20933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:47.371322 20933 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:57:47.371477 20933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:47.521785 20933 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:39339
I20250815 19:57:47.521726 21061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:39339 every 8 connection(s)
I20250815 19:57:47.524804 20933 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:57:47.534058 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 20933
I20250815 19:57:47.534566 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 19:57:47.539939 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--builtin_ntp_servers=127.20.53.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:57:47.551714 21062 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:47.552131 21062 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:47.553130 21062 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:47.555522 20740 ts_manager.cc:194] Registered new tserver with Master: 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339)
I20250815 19:57:47.557600 20740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:46699
W20250815 19:57:47.850333 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:47.850833 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:57:47.851356 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:47.882429 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:57:47.883257 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:57:47.917546 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:57:47.918828 21066 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:57:47.920300 21066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:57:47.932107 21072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:57:48.561065 21062 heartbeater.cc:499] Master 127.20.53.126:45645 was elected leader, sending a full tablet report...
W20250815 19:57:47.933604 21073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:49.121246 21075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:49.123605 21074 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1186 milliseconds
I20250815 19:57:49.123735 21066 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:57:49.124931 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:57:49.127568 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:57:49.129004 21066 hybrid_clock.cc:648] HybridClock initialized: now 1755287869128972 us; error 42 us; skew 500 ppm
I20250815 19:57:49.129773 21066 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:57:49.136010 21066 webserver.cc:480] Webserver started at http://127.20.53.67:43451/ using document root <none> and password file <none>
I20250815 19:57:49.137038 21066 fs_manager.cc:362] Metadata directory not provided
I20250815 19:57:49.137341 21066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:57:49.137780 21066 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:57:49.142077 21066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "43056be5bcc445fcbf64e70bfd7f0377"
format_stamp: "Formatted at 2025-08-15 19:57:49 on dist-test-slave-d4bs"
I20250815 19:57:49.143152 21066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "43056be5bcc445fcbf64e70bfd7f0377"
format_stamp: "Formatted at 2025-08-15 19:57:49 on dist-test-slave-d4bs"
I20250815 19:57:49.151906 21066 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250815 19:57:49.158407 21082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:49.159449 21066 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250815 19:57:49.159744 21066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "43056be5bcc445fcbf64e70bfd7f0377"
format_stamp: "Formatted at 2025-08-15 19:57:49 on dist-test-slave-d4bs"
I20250815 19:57:49.160030 21066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:57:49.209854 21066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:57:49.211234 21066 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:57:49.211629 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:57:49.214032 21066 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:57:49.218943 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:57:49.219156 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:49.219373 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:57:49.219551 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:49.369738 21066 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:43861
I20250815 19:57:49.369822 21194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:43861 every 8 connection(s)
I20250815 19:57:49.372244 21066 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:57:49.382308 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21066
I20250815 19:57:49.382817 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 19:57:49.388975 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.68:0
--local_ip_for_outbound_sockets=127.20.53.68
--webserver_interface=127.20.53.68
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--builtin_ntp_servers=127.20.53.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:57:49.395854 21195 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:49.396314 21195 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:49.397711 21195 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:49.400544 20739 ts_manager.cc:194] Registered new tserver with Master: 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861)
I20250815 19:57:49.401782 20739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:36637
W20250815 19:57:49.701069 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:49.701566 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:57:49.702041 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:49.732282 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:57:49.733119 21199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.68
I20250815 19:57:49.766824 21199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.53.68
--webserver_port=0
--tserver_master_addrs=127.20.53.126:45645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.68
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:57:49.768042 21199 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:57:49.769594 21199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:57:49.782220 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:57:50.405319 21195 heartbeater.cc:499] Master 127.20.53.126:45645 was elected leader, sending a full tablet report...
W20250815 19:57:49.782618 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:51.136248 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:51.139446 21208 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1355 milliseconds
W20250815 19:57:51.140659 21199 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.358s user 0.403s sys 0.927s
W20250815 19:57:51.141036 21199 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.359s user 0.407s sys 0.927s
I20250815 19:57:51.141333 21199 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:57:51.142853 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:57:51.145903 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:57:51.147729 21199 hybrid_clock.cc:648] HybridClock initialized: now 1755287871147676 us; error 58 us; skew 500 ppm
I20250815 19:57:51.148949 21199 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:57:51.158615 21199 webserver.cc:480] Webserver started at http://127.20.53.68:33299/ using document root <none> and password file <none>
I20250815 19:57:51.160017 21199 fs_manager.cc:362] Metadata directory not provided
I20250815 19:57:51.160348 21199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:57:51.161016 21199 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:57:51.168406 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3a5e499d68814352b6f2acf38fbf75ce"
format_stamp: "Formatted at 2025-08-15 19:57:51 on dist-test-slave-d4bs"
I20250815 19:57:51.170061 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3a5e499d68814352b6f2acf38fbf75ce"
format_stamp: "Formatted at 2025-08-15 19:57:51 on dist-test-slave-d4bs"
I20250815 19:57:51.180213 21199 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20250815 19:57:51.187862 21216 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:51.189096 21199 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 19:57:51.189502 21199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3a5e499d68814352b6f2acf38fbf75ce"
format_stamp: "Formatted at 2025-08-15 19:57:51 on dist-test-slave-d4bs"
I20250815 19:57:51.189980 21199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:57:51.278214 21199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:57:51.279651 21199 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:57:51.280076 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:57:51.282601 21199 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:57:51.286616 21199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:57:51.286796 21199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:51.286983 21199 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:57:51.287106 21199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:51.434054 21199 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.68:38887
I20250815 19:57:51.434162 21329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.68:38887 every 8 connection(s)
I20250815 19:57:51.437197 21199 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 19:57:51.444772 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21199
I20250815 19:57:51.445441 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 19:57:51.459584 21330 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:51.460006 21330 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:51.461019 21330 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:51.463083 20740 ts_manager.cc:194] Registered new tserver with Master: 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68:38887)
I20250815 19:57:51.465099 20740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.68:43421
I20250815 19:57:51.467371 20693 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 19:57:51.508767 20740 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39854:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 19:57:51.591439 20997 tablet_service.cc:1468] Processing CreateTablet for tablet 427bb8f24cdf4d4db6d54450bca2dc96 (DEFAULT_TABLE table=TestTable [id=d35d7cea576f4f2ca53ca41142e53333]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:57:51.593696 20997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 427bb8f24cdf4d4db6d54450bca2dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:57:51.599311 21130 tablet_service.cc:1468] Processing CreateTablet for tablet 427bb8f24cdf4d4db6d54450bca2dc96 (DEFAULT_TABLE table=TestTable [id=d35d7cea576f4f2ca53ca41142e53333]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:57:51.602586 21130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 427bb8f24cdf4d4db6d54450bca2dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:57:51.605144 20864 tablet_service.cc:1468] Processing CreateTablet for tablet 427bb8f24cdf4d4db6d54450bca2dc96 (DEFAULT_TABLE table=TestTable [id=d35d7cea576f4f2ca53ca41142e53333]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:57:51.606952 20864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 427bb8f24cdf4d4db6d54450bca2dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:57:51.618078 21349 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: Bootstrap starting.
I20250815 19:57:51.624675 21349 tablet_bootstrap.cc:654] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: Neither blocks nor log segments found. Creating new log.
I20250815 19:57:51.626806 21349 log.cc:826] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: Log is configured to *not* fsync() on all Append() calls
I20250815 19:57:51.632323 21349 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: No bootstrap required, opened a new log
I20250815 19:57:51.633038 21349 ts_tablet_manager.cc:1397] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20250815 19:57:51.634449 21351 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: Bootstrap starting.
I20250815 19:57:51.638706 21352 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Bootstrap starting.
I20250815 19:57:51.642017 21351 tablet_bootstrap.cc:654] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: Neither blocks nor log segments found. Creating new log.
I20250815 19:57:51.644908 21351 log.cc:826] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: Log is configured to *not* fsync() on all Append() calls
I20250815 19:57:51.645051 21352 tablet_bootstrap.cc:654] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Neither blocks nor log segments found. Creating new log.
I20250815 19:57:51.646951 21352 log.cc:826] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Log is configured to *not* fsync() on all Append() calls
I20250815 19:57:51.650992 21351 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: No bootstrap required, opened a new log
I20250815 19:57:51.651566 21351 ts_tablet_manager.cc:1397] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20250815 19:57:51.663128 21349 raft_consensus.cc:357] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.664312 21349 raft_consensus.cc:383] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:57:51.664692 21349 raft_consensus.cc:738] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8815e8735d824256ae89701ac5baa20d, State: Initialized, Role: FOLLOWER
I20250815 19:57:51.665702 21349 consensus_queue.cc:260] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.666774 21352 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: No bootstrap required, opened a new log
I20250815 19:57:51.667824 21352 ts_tablet_manager.cc:1397] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.014s
I20250815 19:57:51.682755 21349 ts_tablet_manager.cc:1428] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: Time spent starting tablet: real 0.049s user 0.025s sys 0.015s
I20250815 19:57:51.683933 21351 raft_consensus.cc:357] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.684965 21351 raft_consensus.cc:383] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:57:51.685281 21351 raft_consensus.cc:738] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43056be5bcc445fcbf64e70bfd7f0377, State: Initialized, Role: FOLLOWER
I20250815 19:57:51.686791 21351 consensus_queue.cc:260] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.692312 21352 raft_consensus.cc:357] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.693372 21352 raft_consensus.cc:383] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:57:51.693707 21352 raft_consensus.cc:738] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b510897cd8674573b420c4490a85bb14, State: Initialized, Role: FOLLOWER
I20250815 19:57:51.694514 21351 ts_tablet_manager.cc:1428] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: Time spent starting tablet: real 0.043s user 0.028s sys 0.006s
I20250815 19:57:51.694749 21352 consensus_queue.cc:260] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.699039 21352 ts_tablet_manager.cc:1428] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Time spent starting tablet: real 0.031s user 0.020s sys 0.007s
W20250815 19:57:51.776463 20930 tablet.cc:2378] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 19:57:51.784201 21063 tablet.cc:2378] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:57:51.824307 21355 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:57:51.825333 21355 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.827714 21355 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861), b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
I20250815 19:57:51.841491 20884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "8815e8735d824256ae89701ac5baa20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b510897cd8674573b420c4490a85bb14" is_pre_election: true
I20250815 19:57:51.841526 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "8815e8735d824256ae89701ac5baa20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43056be5bcc445fcbf64e70bfd7f0377" is_pre_election: true
I20250815 19:57:51.842404 21150 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8815e8735d824256ae89701ac5baa20d in term 0.
I20250815 19:57:51.842404 20884 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8815e8735d824256ae89701ac5baa20d in term 0.
I20250815 19:57:51.843604 20950 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8815e8735d824256ae89701ac5baa20d, b510897cd8674573b420c4490a85bb14; no voters:
I20250815 19:57:51.844558 21355 raft_consensus.cc:2802] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 19:57:51.844866 21355 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:57:51.845075 21355 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:57:51.849613 21355 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.850991 21355 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [CANDIDATE]: Term 1 election: Requested vote from peers 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861), b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
I20250815 19:57:51.851799 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "8815e8735d824256ae89701ac5baa20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43056be5bcc445fcbf64e70bfd7f0377"
I20250815 19:57:51.852043 20884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "8815e8735d824256ae89701ac5baa20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b510897cd8674573b420c4490a85bb14"
I20250815 19:57:51.852277 21150 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:57:51.852510 20884 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:57:51.857637 21150 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8815e8735d824256ae89701ac5baa20d in term 1.
I20250815 19:57:51.857851 20884 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8815e8735d824256ae89701ac5baa20d in term 1.
I20250815 19:57:51.858722 20953 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377, 8815e8735d824256ae89701ac5baa20d; no voters:
I20250815 19:57:51.859366 21355 raft_consensus.cc:2802] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:57:51.861289 21355 raft_consensus.cc:695] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [term 1 LEADER]: Becoming Leader. State: Replica: 8815e8735d824256ae89701ac5baa20d, State: Running, Role: LEADER
I20250815 19:57:51.862048 21355 consensus_queue.cc:237] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:51.872048 20738 catalog_manager.cc:5582] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8815e8735d824256ae89701ac5baa20d (127.20.53.66). New cstate: current_term: 1 leader_uuid: "8815e8735d824256ae89701ac5baa20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } health_report { overall_health: UNKNOWN } } }
W20250815 19:57:51.879666 21196 tablet.cc:2378] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:57:51.910653 20693 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 19:57:51.914285 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b510897cd8674573b420c4490a85bb14 to finish bootstrapping
I20250815 19:57:51.932868 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8815e8735d824256ae89701ac5baa20d to finish bootstrapping
I20250815 19:57:51.943069 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 43056be5bcc445fcbf64e70bfd7f0377 to finish bootstrapping
I20250815 19:57:51.954025 20693 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250815 19:57:51.957072 20693 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "8815e8735d824256ae89701ac5baa20d"
rpc_addresses {
host: "127.20.53.66"
port: 39339
}
}
ts_infos {
permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377"
rpc_addresses {
host: "127.20.53.67"
port: 43861
}
}
ts_infos {
permanent_uuid: "b510897cd8674573b420c4490a85bb14"
rpc_addresses {
host: "127.20.53.65"
port: 44477
}
}
I20250815 19:57:52.288931 21355 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 19:57:52.304575 21355 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P 8815e8735d824256ae89701ac5baa20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 19:57:52.305533 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 20933
W20250815 19:57:52.330191 20726 connection.cc:537] server connection from 127.20.53.66:46699 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250815 19:57:52.330953 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 20707
I20250815 19:57:52.357920 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:45645
--webserver_interface=127.20.53.126
--webserver_port=37419
--builtin_ntp_servers=127.20.53.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:45645 with env {}
I20250815 19:57:52.468132 21330 heartbeater.cc:499] Master 127.20.53.126:45645 was elected leader, sending a full tablet report...
W20250815 19:57:52.471801 21330 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:45645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:45645: connect: Connection refused (error 111)
W20250815 19:57:52.660996 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:52.661586 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:57:52.662047 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:52.691557 21368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:57:52.691874 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:57:52.692117 21368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:57:52.692359 21368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250815 19:57:52.708781 21195 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:45645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:45645: connect: Connection refused (error 111)
I20250815 19:57:52.726992 21368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:45645
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=37419
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:57:52.728237 21368 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:57:52.729773 21368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:57:52.739440 21376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:53.313405 20929 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:45645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:45645: connect: Connection refused (error 111)
I20250815 19:57:53.466956 21383 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:57:53.467373 21383 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:53.469528 21383 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339), b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
W20250815 19:57:53.474581 21086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111)
W20250815 19:57:53.486580 21086 leader_election.cc:336] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339): Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111)
I20250815 19:57:53.488036 20884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "43056be5bcc445fcbf64e70bfd7f0377" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b510897cd8674573b420c4490a85bb14" is_pre_election: true
I20250815 19:57:53.489472 21083 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377; no voters: 8815e8735d824256ae89701ac5baa20d, b510897cd8674573b420c4490a85bb14
I20250815 19:57:53.490443 21383 raft_consensus.cc:2747] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250815 19:57:53.801966 21390 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8815e8735d824256ae89701ac5baa20d)
I20250815 19:57:53.802887 21390 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:53.806725 21390 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339), 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861)
W20250815 19:57:53.811444 20820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111)
W20250815 19:57:53.816174 20820 leader_election.cc:336] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339): Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111)
I20250815 19:57:53.821693 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "b510897cd8674573b420c4490a85bb14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43056be5bcc445fcbf64e70bfd7f0377" is_pre_election: true
I20250815 19:57:53.822335 21150 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b510897cd8674573b420c4490a85bb14 in term 1.
I20250815 19:57:53.823369 20820 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377, b510897cd8674573b420c4490a85bb14; no voters: 8815e8735d824256ae89701ac5baa20d
I20250815 19:57:53.824007 21390 raft_consensus.cc:2802] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 19:57:53.824366 21390 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8815e8735d824256ae89701ac5baa20d)
I20250815 19:57:53.824690 21390 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:57:53.847199 21390 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:53.849577 21390 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 2 election: Requested vote from peers 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339), 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861)
I20250815 19:57:53.852394 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "b510897cd8674573b420c4490a85bb14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43056be5bcc445fcbf64e70bfd7f0377"
I20250815 19:57:53.852867 21150 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 1 FOLLOWER]: Advancing to term 2
W20250815 19:57:53.854120 20820 leader_election.cc:336] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339): Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111)
W20250815 19:57:52.741104 21377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:57:53.941265 21378 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 19:57:53.942740 21379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:57:53.942761 21368 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:57:53.945905 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:57:53.948915 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:57:53.949532 21150 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b510897cd8674573b420c4490a85bb14 in term 2.
I20250815 19:57:53.950343 21368 hybrid_clock.cc:648] HybridClock initialized: now 1755287873950297 us; error 48 us; skew 500 ppm
I20250815 19:57:53.950766 20820 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377, b510897cd8674573b420c4490a85bb14; no voters: 8815e8735d824256ae89701ac5baa20d
I20250815 19:57:53.951304 21368 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:57:53.951726 21390 raft_consensus.cc:2802] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 19:57:53.953783 21390 raft_consensus.cc:695] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 LEADER]: Becoming Leader. State: Replica: b510897cd8674573b420c4490a85bb14, State: Running, Role: LEADER
I20250815 19:57:53.954901 21390 consensus_queue.cc:237] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:53.960958 21368 webserver.cc:480] Webserver started at http://127.20.53.126:37419/ using document root <none> and password file <none>
I20250815 19:57:53.961864 21368 fs_manager.cc:362] Metadata directory not provided
I20250815 19:57:53.962076 21368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:57:53.969862 21368 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250815 19:57:53.974555 21400 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:57:53.975664 21368 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250815 19:57:53.975996 21368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "37aa504c6787485d991e0b1d562f738f"
format_stamp: "Formatted at 2025-08-15 19:57:42 on dist-test-slave-d4bs"
I20250815 19:57:53.978277 21368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:57:54.026041 21368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:57:54.027551 21368 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:57:54.028090 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:57:54.099484 21368 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:45645
I20250815 19:57:54.099566 21451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:45645 every 8 connection(s)
I20250815 19:57:54.102409 21368 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:57:54.106441 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21368
I20250815 19:57:54.107024 20693 kudu-admin-test.cc:735] Forcing unsafe config change on tserver b510897cd8674573b420c4490a85bb14
I20250815 19:57:54.116199 21452 sys_catalog.cc:263] Verifying existing consensus state
I20250815 19:57:54.120820 21452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Bootstrap starting.
I20250815 19:57:54.157964 21452 log.cc:826] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Log is configured to *not* fsync() on all Append() calls
I20250815 19:57:54.180037 21452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:57:54.180884 21452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Bootstrap complete.
I20250815 19:57:54.201678 21452 raft_consensus.cc:357] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:54.203673 21452 raft_consensus.cc:738] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aa504c6787485d991e0b1d562f738f, State: Initialized, Role: FOLLOWER
I20250815 19:57:54.204449 21452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:54.204977 21452 raft_consensus.cc:397] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:57:54.205263 21452 raft_consensus.cc:491] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:57:54.205601 21452 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:57:54.253415 21452 raft_consensus.cc:513] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:54.254221 21452 leader_election.cc:304] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37aa504c6787485d991e0b1d562f738f; no voters:
I20250815 19:57:54.256628 21452 leader_election.cc:290] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 19:57:54.256932 21456 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 2 FOLLOWER]: Leader election won for term 2
I20250815 19:57:54.260829 21456 raft_consensus.cc:695] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [term 2 LEADER]: Becoming Leader. State: Replica: 37aa504c6787485d991e0b1d562f738f, State: Running, Role: LEADER
I20250815 19:57:54.261711 21456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } }
I20250815 19:57:54.263082 21452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:57:54.272507 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37aa504c6787485d991e0b1d562f738f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } } }
I20250815 19:57:54.272547 21458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37aa504c6787485d991e0b1d562f738f. Latest consensus state: current_term: 2 leader_uuid: "37aa504c6787485d991e0b1d562f738f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37aa504c6787485d991e0b1d562f738f" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 45645 } } }
I20250815 19:57:54.273483 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: This master's current role is: LEADER
I20250815 19:57:54.273523 21458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f [sys.catalog]: This master's current role is: LEADER
I20250815 19:57:54.282153 21463 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:57:54.299670 21463 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d35d7cea576f4f2ca53ca41142e53333]
I20250815 19:57:54.311446 21463 tablet_loader.cc:96] loaded metadata for tablet 427bb8f24cdf4d4db6d54450bca2dc96 (table TestTable [id=d35d7cea576f4f2ca53ca41142e53333])
I20250815 19:57:54.313174 21463 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:57:54.318184 21463 catalog_manager.cc:1261] Loaded cluster ID: d5b2005f5c0d4b81a05b71e2e6daa4bc
I20250815 19:57:54.318563 21463 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:57:54.326293 21463 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:57:54.331130 21463 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 37aa504c6787485d991e0b1d562f738f: Loaded TSK: 0
I20250815 19:57:54.333524 21463 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 19:57:54.353367 20820 consensus_peers.cc:489] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 -> Peer 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339): Couldn't send request to peer 8815e8735d824256ae89701ac5baa20d. Status: Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 19:57:54.406621 21150 raft_consensus.cc:1273] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Refusing update from remote peer b510897cd8674573b420c4490a85bb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 19:57:54.408067 21390 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 19:57:54.455236 21195 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:54.464149 21417 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" instance_seqno: 1755287869335583) as {username='slave'} at 127.20.53.67:58717; Asking this server to re-register.
I20250815 19:57:54.466095 21195 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:54.466750 21195 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:54.469465 20929 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:54.471794 21416 ts_manager.cc:194] Registered new tserver with Master: 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861)
I20250815 19:57:54.474078 21415 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b510897cd8674573b420c4490a85bb14" instance_seqno: 1755287865470740) as {username='slave'} at 127.20.53.65:45917; Asking this server to re-register.
I20250815 19:57:54.475710 20929 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:54.476653 20929 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:54.477602 21416 catalog_manager.cc:5582] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 reported cstate change: term changed from 1 to 2, leader changed from 8815e8735d824256ae89701ac5baa20d (127.20.53.66) to b510897cd8674573b420c4490a85bb14 (127.20.53.65). New cstate: current_term: 2 leader_uuid: "b510897cd8674573b420c4490a85bb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } }
I20250815 19:57:54.479805 21415 ts_manager.cc:194] Registered new tserver with Master: b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
I20250815 19:57:54.522068 21330 heartbeater.cc:344] Connected to a master server at 127.20.53.126:45645
I20250815 19:57:54.522526 21330 heartbeater.cc:499] Master 127.20.53.126:45645 was elected leader, sending a full tablet report...
I20250815 19:57:54.534417 21417 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" instance_seqno: 1755287871398391) as {username='slave'} at 127.20.53.68:40777; Asking this server to re-register.
I20250815 19:57:54.537093 21330 heartbeater.cc:461] Registering TS with master...
I20250815 19:57:54.537701 21330 heartbeater.cc:507] Master 127.20.53.126:45645 requested a full tablet report, sending...
I20250815 19:57:54.540256 21415 ts_manager.cc:194] Registered new tserver with Master: 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68:38887)
W20250815 19:57:54.541589 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:57:54.542419 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:57:54.577478 21454 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 19:57:56.023584 21488 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 21454
W20250815 19:57:56.024173 21488 kernel_stack_watchdog.cc:198] Thread 21454 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:57:56.269542 21454 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.651s user 0.523s sys 1.099s
W20250815 19:57:56.401317 21454 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.783s user 0.527s sys 1.109s
I20250815 19:57:56.443460 20884 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "b510897cd8674573b420c4490a85bb14"
tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377"
}
peers {
permanent_uuid: "b510897cd8674573b420c4490a85bb14"
}
}
from {username='slave'} at 127.0.0.1:33540
W20250815 19:57:56.444684 20884 raft_consensus.cc:2216] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true
I20250815 19:57:56.445611 20884 raft_consensus.cc:3053] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 LEADER]: Stepping down as leader of term 2
I20250815 19:57:56.445833 20884 raft_consensus.cc:738] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b510897cd8674573b420c4490a85bb14, State: Running, Role: LEADER
I20250815 19:57:56.446468 20884 consensus_queue.cc:260] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:56.447419 20884 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 2 FOLLOWER]: Advancing to term 3
W20250815 19:57:57.225869 21488 debug-util.cc:398] Leaking SignalData structure 0x7b08000379e0 after lost signal to thread 21454
I20250815 19:57:57.867940 21509 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b510897cd8674573b420c4490a85bb14)
I20250815 19:57:57.868667 21509 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } }
I20250815 19:57:57.870472 21509 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339), b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
I20250815 19:57:57.871727 20884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "43056be5bcc445fcbf64e70bfd7f0377" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b510897cd8674573b420c4490a85bb14" is_pre_election: true
W20250815 19:57:57.875826 21086 leader_election.cc:336] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8815e8735d824256ae89701ac5baa20d (127.20.53.66:39339): Network error: Client connection negotiation failed: client connection to 127.20.53.66:39339: connect: Connection refused (error 111)
I20250815 19:57:57.876154 21086 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377; no voters: 8815e8735d824256ae89701ac5baa20d, b510897cd8674573b420c4490a85bb14
I20250815 19:57:57.876809 21509 raft_consensus.cc:2747] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 19:57:58.028019 21512 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250815 19:57:58.028398 21512 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true
I20250815 19:57:58.029399 21512 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861)
I20250815 19:57:58.030689 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "b510897cd8674573b420c4490a85bb14" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "43056be5bcc445fcbf64e70bfd7f0377" is_pre_election: true
I20250815 19:57:58.031219 21150 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b510897cd8674573b420c4490a85bb14 in term 2.
I20250815 19:57:58.032104 20820 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377, b510897cd8674573b420c4490a85bb14; no voters:
I20250815 19:57:58.032702 21512 raft_consensus.cc:2802] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250815 19:57:58.032967 21512 raft_consensus.cc:491] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250815 19:57:58.033216 21512 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 3 FOLLOWER]: Advancing to term 4
I20250815 19:57:58.037402 21512 raft_consensus.cc:513] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true
I20250815 19:57:58.038385 21512 leader_election.cc:290] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 4 election: Requested vote from peers 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67:43861)
I20250815 19:57:58.039343 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96" candidate_uuid: "b510897cd8674573b420c4490a85bb14" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "43056be5bcc445fcbf64e70bfd7f0377"
I20250815 19:57:58.039709 21150 raft_consensus.cc:3058] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 2 FOLLOWER]: Advancing to term 4
I20250815 19:57:58.043977 21150 raft_consensus.cc:2466] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b510897cd8674573b420c4490a85bb14 in term 4.
I20250815 19:57:58.044863 20820 leader_election.cc:304] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43056be5bcc445fcbf64e70bfd7f0377, b510897cd8674573b420c4490a85bb14; no voters:
I20250815 19:57:58.045449 21512 raft_consensus.cc:2802] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 19:57:58.046257 21512 raft_consensus.cc:695] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 4 LEADER]: Becoming Leader. State: Replica: b510897cd8674573b420c4490a85bb14, State: Running, Role: LEADER
I20250815 19:57:58.046813 21512 consensus_queue.cc:237] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true
I20250815 19:57:58.052120 21415 catalog_manager.cc:5582] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67), VOTER b510897cd8674573b420c4490a85bb14 (127.20.53.65). New cstate: current_term: 4 leader_uuid: "b510897cd8674573b420c4490a85bb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8815e8735d824256ae89701ac5baa20d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39339 } } peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true }
I20250815 19:57:58.534058 21150 raft_consensus.cc:1273] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Refusing update from remote peer b510897cd8674573b420c4490a85bb14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250815 19:57:58.535202 21512 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250815 19:57:58.542331 21513 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8815e8735d824256ae89701ac5baa20d (127.20.53.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true }
I20250815 19:57:58.543311 21150 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8815e8735d824256ae89701ac5baa20d (127.20.53.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } unsafe_config_change: true }
I20250815 19:57:58.554042 21415 catalog_manager.cc:5582] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 reported cstate change: config changed from index -1 to 3, VOTER 8815e8735d824256ae89701ac5baa20d (127.20.53.66) evicted, no longer has a pending config: VOTER 43056be5bcc445fcbf64e70bfd7f0377 (127.20.53.67), VOTER b510897cd8674573b420c4490a85bb14 (127.20.53.65). New cstate: current_term: 4 leader_uuid: "b510897cd8674573b420c4490a85bb14" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250815 19:57:58.561463 21415 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 427bb8f24cdf4d4db6d54450bca2dc96 on TS 8815e8735d824256ae89701ac5baa20d: Not found: failed to reset TS proxy: Could not find TS for UUID 8815e8735d824256ae89701ac5baa20d
I20250815 19:57:58.577198 20884 consensus_queue.cc:237] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: NON_VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: true } } unsafe_config_change: true
I20250815 19:57:58.581640 21150 raft_consensus.cc:1273] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Refusing update from remote peer b510897cd8674573b420c4490a85bb14: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250815 19:57:58.583253 21512 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250815 19:57:58.588692 21513 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: NON_VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: true } } unsafe_config_change: true }
I20250815 19:57:58.590417 21150 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: NON_VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: true } } unsafe_config_change: true }
W20250815 19:57:58.592051 20819 consensus_peers.cc:489] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 -> Peer 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68:38887): Couldn't send request to peer 3a5e499d68814352b6f2acf38fbf75ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 427bb8f24cdf4d4db6d54450bca2dc96. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:57:58.597265 21401 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 427bb8f24cdf4d4db6d54450bca2dc96 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 19:57:58.600852 21416 catalog_manager.cc:5582] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 reported cstate change: config changed from index 3 to 5, NON_VOTER 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) added. New cstate: current_term: 4 leader_uuid: "b510897cd8674573b420c4490a85bb14" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: NON_VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250815 19:57:58.613901 21402 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 427bb8f24cdf4d4db6d54450bca2dc96 on TS 8815e8735d824256ae89701ac5baa20d failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8815e8735d824256ae89701ac5baa20d
I20250815 19:57:58.970822 21528 ts_tablet_manager.cc:927] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Initiating tablet copy from peer b510897cd8674573b420c4490a85bb14 (127.20.53.65:44477)
I20250815 19:57:58.973114 21528 tablet_copy_client.cc:323] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: tablet copy: Beginning tablet copy session from remote peer at address 127.20.53.65:44477
I20250815 19:57:58.982897 20904 tablet_copy_service.cc:140] P b510897cd8674573b420c4490a85bb14: Received BeginTabletCopySession request for tablet 427bb8f24cdf4d4db6d54450bca2dc96 from peer 3a5e499d68814352b6f2acf38fbf75ce ({username='slave'} at 127.20.53.68:34395)
I20250815 19:57:58.983304 20904 tablet_copy_service.cc:161] P b510897cd8674573b420c4490a85bb14: Beginning new tablet copy session on tablet 427bb8f24cdf4d4db6d54450bca2dc96 from peer 3a5e499d68814352b6f2acf38fbf75ce at {username='slave'} at 127.20.53.68:34395: session id = 3a5e499d68814352b6f2acf38fbf75ce-427bb8f24cdf4d4db6d54450bca2dc96
I20250815 19:57:58.988389 20904 tablet_copy_source_session.cc:215] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 19:57:58.993279 21528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 427bb8f24cdf4d4db6d54450bca2dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:57:59.009999 21528 tablet_copy_client.cc:806] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: tablet copy: Starting download of 0 data blocks...
I20250815 19:57:59.010481 21528 tablet_copy_client.cc:670] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: tablet copy: Starting download of 1 WAL segments...
I20250815 19:57:59.014037 21528 tablet_copy_client.cc:538] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 19:57:59.019980 21528 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Bootstrap starting.
I20250815 19:57:59.031597 21528 log.cc:826] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Log is configured to *not* fsync() on all Append() calls
I20250815 19:57:59.041731 21528 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:57:59.042366 21528 tablet_bootstrap.cc:492] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Bootstrap complete.
I20250815 19:57:59.042908 21528 ts_tablet_manager.cc:1397] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.008s
I20250815 19:57:59.059691 21528 raft_consensus.cc:357] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: NON_VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: true } } unsafe_config_change: true
I20250815 19:57:59.060545 21528 raft_consensus.cc:738] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 3a5e499d68814352b6f2acf38fbf75ce, State: Initialized, Role: LEARNER
I20250815 19:57:59.061285 21528 consensus_queue.cc:260] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: NON_VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: true } } unsafe_config_change: true
I20250815 19:57:59.064225 21528 ts_tablet_manager.cc:1428] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20250815 19:57:59.065922 20904 tablet_copy_service.cc:342] P b510897cd8674573b420c4490a85bb14: Request end of tablet copy session 3a5e499d68814352b6f2acf38fbf75ce-427bb8f24cdf4d4db6d54450bca2dc96 received from {username='slave'} at 127.20.53.68:34395
I20250815 19:57:59.066258 20904 tablet_copy_service.cc:434] P b510897cd8674573b420c4490a85bb14: ending tablet copy session 3a5e499d68814352b6f2acf38fbf75ce-427bb8f24cdf4d4db6d54450bca2dc96 on tablet 427bb8f24cdf4d4db6d54450bca2dc96 with peer 3a5e499d68814352b6f2acf38fbf75ce
I20250815 19:57:59.347555 21285 raft_consensus.cc:1215] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
I20250815 19:57:59.779695 21534 raft_consensus.cc:1062] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14: attempting to promote NON_VOTER 3a5e499d68814352b6f2acf38fbf75ce to VOTER
W20250815 19:57:59.779925 21402 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 427bb8f24cdf4d4db6d54450bca2dc96 on TS 8815e8735d824256ae89701ac5baa20d failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8815e8735d824256ae89701ac5baa20d
I20250815 19:57:59.781378 21534 consensus_queue.cc:237] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: false } } unsafe_config_change: true
I20250815 19:57:59.786007 21150 raft_consensus.cc:1273] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Refusing update from remote peer b510897cd8674573b420c4490a85bb14: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 19:57:59.786203 21285 raft_consensus.cc:1273] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce [term 4 LEARNER]: Refusing update from remote peer b510897cd8674573b420c4490a85bb14: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 19:57:59.787053 21527 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250815 19:57:59.787878 21533 consensus_queue.cc:1035] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250815 19:57:59.793923 21527 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: false } } unsafe_config_change: true }
I20250815 19:57:59.795320 21150 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P 43056be5bcc445fcbf64e70bfd7f0377 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: false } } unsafe_config_change: true }
I20250815 19:57:59.797291 21285 raft_consensus.cc:2953] T 427bb8f24cdf4d4db6d54450bca2dc96 P 3a5e499d68814352b6f2acf38fbf75ce [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: false } } unsafe_config_change: true }
I20250815 19:57:59.806434 21416 catalog_manager.cc:5582] T 427bb8f24cdf4d4db6d54450bca2dc96 P b510897cd8674573b420c4490a85bb14 reported cstate change: config changed from index 5 to 6, 3a5e499d68814352b6f2acf38fbf75ce (127.20.53.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "b510897cd8674573b420c4490a85bb14" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 43861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b510897cd8674573b420c4490a85bb14" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 38887 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250815 19:57:59.816891 20693 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250815 19:57:59.838631 20693 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "427bb8f24cdf4d4db6d54450bca2dc96"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "43056be5bcc445fcbf64e70bfd7f0377"
rpc_addresses {
host: "127.20.53.67"
port: 43861
}
}
ts_infos {
permanent_uuid: "b510897cd8674573b420c4490a85bb14"
rpc_addresses {
host: "127.20.53.65"
port: 44477
}
}
ts_infos {
permanent_uuid: "3a5e499d68814352b6f2acf38fbf75ce"
rpc_addresses {
host: "127.20.53.68"
port: 38887
}
}
I20250815 19:57:59.841949 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 20800
I20250815 19:57:59.867933 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21066
I20250815 19:57:59.892565 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21199
I20250815 19:57:59.917169 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21368
2025-08-15T19:57:59Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (20253 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250815 19:57:59.978171 20693 test_util.cc:276] Using random seed: -829387616
I20250815 19:57:59.984028 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 19:57:59.984196 20693 ts_itest-base.cc:116] --------------
I20250815 19:57:59.984522 20693 ts_itest-base.cc:117] 3 tablet servers
I20250815 19:57:59.984701 20693 ts_itest-base.cc:118] 3 replicas per TS
I20250815 19:57:59.984889 20693 ts_itest-base.cc:119] --------------
2025-08-15T19:58:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T19:58:00Z Disabled control of system clock
I20250815 19:58:00.021339 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:42149
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:42149
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250815 19:58:00.316001 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:00.316622 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:00.317097 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:00.348276 21554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:58:00.348623 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:00.348888 21554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:58:00.349118 21554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:58:00.384270 21554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:42149
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:00.385650 21554 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:00.387225 21554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:00.398303 21560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:00.399289 21561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:01.563668 21562 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 19:58:01.565152 21563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:01.565169 21554 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:01.568569 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:01.571697 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:01.573104 21554 hybrid_clock.cc:648] HybridClock initialized: now 1755287881573061 us; error 55 us; skew 500 ppm
I20250815 19:58:01.573915 21554 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:01.581017 21554 webserver.cc:480] Webserver started at http://127.20.53.126:34711/ using document root <none> and password file <none>
I20250815 19:58:01.581918 21554 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:01.582118 21554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:01.582595 21554 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:01.587227 21554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "87d1ca8ff571429b977933fa527b5d8e"
format_stamp: "Formatted at 2025-08-15 19:58:01 on dist-test-slave-d4bs"
I20250815 19:58:01.588361 21554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "87d1ca8ff571429b977933fa527b5d8e"
format_stamp: "Formatted at 2025-08-15 19:58:01 on dist-test-slave-d4bs"
I20250815 19:58:01.595588 21554 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250815 19:58:01.601398 21570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:01.602519 21554 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 19:58:01.602810 21554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "87d1ca8ff571429b977933fa527b5d8e"
format_stamp: "Formatted at 2025-08-15 19:58:01 on dist-test-slave-d4bs"
I20250815 19:58:01.603106 21554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:01.665503 21554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:01.666957 21554 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:01.667382 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:01.737850 21554 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:42149
I20250815 19:58:01.737942 21621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:42149 every 8 connection(s)
I20250815 19:58:01.740574 21554 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:58:01.746006 21622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:01.748801 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21554
I20250815 19:58:01.749295 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 19:58:01.767894 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e: Bootstrap starting.
I20250815 19:58:01.773947 21622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:01.775609 21622 log.cc:826] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:01.780242 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e: No bootstrap required, opened a new log
I20250815 19:58:01.797487 21622 raft_consensus.cc:357] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d1ca8ff571429b977933fa527b5d8e" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 42149 } }
I20250815 19:58:01.798148 21622 raft_consensus.cc:383] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:01.798420 21622 raft_consensus.cc:738] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d1ca8ff571429b977933fa527b5d8e, State: Initialized, Role: FOLLOWER
I20250815 19:58:01.799187 21622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d1ca8ff571429b977933fa527b5d8e" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 42149 } }
I20250815 19:58:01.799764 21622 raft_consensus.cc:397] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:01.800127 21622 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:01.800556 21622 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:01.804641 21622 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d1ca8ff571429b977933fa527b5d8e" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 42149 } }
I20250815 19:58:01.805382 21622 leader_election.cc:304] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d1ca8ff571429b977933fa527b5d8e; no voters:
I20250815 19:58:01.806998 21622 leader_election.cc:290] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:01.807678 21627 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:01.809965 21627 raft_consensus.cc:695] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [term 1 LEADER]: Becoming Leader. State: Replica: 87d1ca8ff571429b977933fa527b5d8e, State: Running, Role: LEADER
I20250815 19:58:01.811126 21622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:58:01.810897 21627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d1ca8ff571429b977933fa527b5d8e" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 42149 } }
I20250815 19:58:01.822806 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d1ca8ff571429b977933fa527b5d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d1ca8ff571429b977933fa527b5d8e" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 42149 } } }
I20250815 19:58:01.824327 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d1ca8ff571429b977933fa527b5d8e. Latest consensus state: current_term: 1 leader_uuid: "87d1ca8ff571429b977933fa527b5d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d1ca8ff571429b977933fa527b5d8e" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 42149 } } }
I20250815 19:58:01.824936 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:01.825153 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:01.828603 21637 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:58:01.838912 21637 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:58:01.853106 21637 catalog_manager.cc:1349] Generated new cluster ID: 5b9887ffff3e41128ec7aef6e374c5e5
I20250815 19:58:01.853435 21637 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:58:01.868287 21637 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 19:58:01.869706 21637 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:58:01.891696 21637 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 87d1ca8ff571429b977933fa527b5d8e: Generated new TSK 0
I20250815 19:58:01.892560 21637 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 19:58:01.902263 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:42149
--builtin_ntp_servers=127.20.53.84:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250815 19:58:02.220449 21646 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 19:58:02.221133 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:02.221401 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:02.221880 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:02.252887 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:02.253777 21646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:58:02.289407 21646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:42149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:02.290791 21646 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:02.292424 21646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:02.305343 21652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:02.307186 21653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:03.877311 21655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:03.879878 21654 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1570 milliseconds
I20250815 19:58:03.880007 21646 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:03.881376 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:03.884199 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:03.885625 21646 hybrid_clock.cc:648] HybridClock initialized: now 1755287883885578 us; error 58 us; skew 500 ppm
I20250815 19:58:03.886440 21646 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:03.893174 21646 webserver.cc:480] Webserver started at http://127.20.53.65:42557/ using document root <none> and password file <none>
I20250815 19:58:03.894088 21646 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:03.894282 21646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:03.894732 21646 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:03.899212 21646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
format_stamp: "Formatted at 2025-08-15 19:58:03 on dist-test-slave-d4bs"
I20250815 19:58:03.900313 21646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
format_stamp: "Formatted at 2025-08-15 19:58:03 on dist-test-slave-d4bs"
I20250815 19:58:03.907382 21646 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250815 19:58:03.912966 21662 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:03.913965 21646 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 19:58:03.914271 21646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
format_stamp: "Formatted at 2025-08-15 19:58:03 on dist-test-slave-d4bs"
I20250815 19:58:03.914580 21646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:03.977610 21646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:03.978976 21646 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:03.979409 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:03.981822 21646 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:03.985715 21646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:03.985929 21646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:03.986153 21646 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:03.986297 21646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:04.118567 21646 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:41249
I20250815 19:58:04.118666 21774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:41249 every 8 connection(s)
I20250815 19:58:04.121057 21646 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:58:04.126962 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21646
I20250815 19:58:04.127342 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 19:58:04.133214 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:42149
--builtin_ntp_servers=127.20.53.84:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 19:58:04.142840 21775 heartbeater.cc:344] Connected to a master server at 127.20.53.126:42149
I20250815 19:58:04.143285 21775 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:04.144322 21775 heartbeater.cc:507] Master 127.20.53.126:42149 requested a full tablet report, sending...
I20250815 19:58:04.147089 21587 ts_manager.cc:194] Registered new tserver with Master: 1c4aad0f9f0648d793b41eba69e8a6ee (127.20.53.65:41249)
I20250815 19:58:04.149147 21587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:49935
W20250815 19:58:04.433403 21779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 19:58:04.434053 21779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:04.434315 21779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:04.434793 21779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:04.465018 21779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:04.465895 21779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:58:04.498817 21779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:42149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:04.500098 21779 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:04.501721 21779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:04.514215 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:05.152457 21775 heartbeater.cc:499] Master 127.20.53.126:42149 was elected leader, sending a full tablet report...
W20250815 19:58:04.516134 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:04.518146 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:05.672672 21787 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 19:58:05.672724 21779 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:05.676476 21779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:05.678660 21779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:05.679983 21779 hybrid_clock.cc:648] HybridClock initialized: now 1755287885679968 us; error 41 us; skew 500 ppm
I20250815 19:58:05.680796 21779 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:05.686759 21779 webserver.cc:480] Webserver started at http://127.20.53.66:33965/ using document root <none> and password file <none>
I20250815 19:58:05.687665 21779 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:05.687846 21779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:05.688266 21779 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:05.692687 21779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "76e9fcc45b244962b2347a78e4c82b3d"
format_stamp: "Formatted at 2025-08-15 19:58:05 on dist-test-slave-d4bs"
I20250815 19:58:05.693814 21779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "76e9fcc45b244962b2347a78e4c82b3d"
format_stamp: "Formatted at 2025-08-15 19:58:05 on dist-test-slave-d4bs"
I20250815 19:58:05.700762 21779 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250815 19:58:05.706082 21795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:05.707175 21779 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250815 19:58:05.707476 21779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "76e9fcc45b244962b2347a78e4c82b3d"
format_stamp: "Formatted at 2025-08-15 19:58:05 on dist-test-slave-d4bs"
I20250815 19:58:05.707788 21779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:05.757884 21779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:05.759302 21779 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:05.759682 21779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:05.762111 21779 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:05.766038 21779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:05.766222 21779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:05.766440 21779 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:05.766572 21779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:05.897526 21779 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:41475
I20250815 19:58:05.897643 21907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:41475 every 8 connection(s)
I20250815 19:58:05.900080 21779 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:58:05.905043 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21779
I20250815 19:58:05.905524 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 19:58:05.911039 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:42149
--builtin_ntp_servers=127.20.53.84:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 19:58:05.925573 21908 heartbeater.cc:344] Connected to a master server at 127.20.53.126:42149
I20250815 19:58:05.926116 21908 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:05.927536 21908 heartbeater.cc:507] Master 127.20.53.126:42149 requested a full tablet report, sending...
I20250815 19:58:05.930133 21587 ts_manager.cc:194] Registered new tserver with Master: 76e9fcc45b244962b2347a78e4c82b3d (127.20.53.66:41475)
I20250815 19:58:05.931424 21587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:58957
W20250815 19:58:06.217367 21912 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 19:58:06.217988 21912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:06.218261 21912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:06.218741 21912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:06.249347 21912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:06.250344 21912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:58:06.284382 21912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:42149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:06.285763 21912 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:06.287753 21912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:06.298877 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:06.935273 21908 heartbeater.cc:499] Master 127.20.53.126:42149 was elected leader, sending a full tablet report...
W20250815 19:58:06.300572 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:07.464942 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:07.466890 21920 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1162 milliseconds
I20250815 19:58:07.467027 21912 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:07.468164 21912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:07.470913 21912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:07.472369 21912 hybrid_clock.cc:648] HybridClock initialized: now 1755287887472343 us; error 35 us; skew 500 ppm
I20250815 19:58:07.473176 21912 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:07.479856 21912 webserver.cc:480] Webserver started at http://127.20.53.67:38917/ using document root <none> and password file <none>
I20250815 19:58:07.480850 21912 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:07.481053 21912 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:07.481519 21912 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:07.485908 21912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "711512a9a63e438c9ffbda240305be04"
format_stamp: "Formatted at 2025-08-15 19:58:07 on dist-test-slave-d4bs"
I20250815 19:58:07.487018 21912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "711512a9a63e438c9ffbda240305be04"
format_stamp: "Formatted at 2025-08-15 19:58:07 on dist-test-slave-d4bs"
I20250815 19:58:07.494316 21912 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250815 19:58:07.499979 21928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:07.501085 21912 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250815 19:58:07.501394 21912 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "711512a9a63e438c9ffbda240305be04"
format_stamp: "Formatted at 2025-08-15 19:58:07 on dist-test-slave-d4bs"
I20250815 19:58:07.501717 21912 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:07.552803 21912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:07.554204 21912 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:07.554639 21912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:07.557106 21912 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:07.561060 21912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:07.561255 21912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:07.561523 21912 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:07.561679 21912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:07.696305 21912 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:37853
I20250815 19:58:07.696403 22040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:37853 every 8 connection(s)
I20250815 19:58:07.698777 21912 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:58:07.705389 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 21912
I20250815 19:58:07.705870 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 19:58:07.720865 22041 heartbeater.cc:344] Connected to a master server at 127.20.53.126:42149
I20250815 19:58:07.721324 22041 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:07.722355 22041 heartbeater.cc:507] Master 127.20.53.126:42149 requested a full tablet report, sending...
I20250815 19:58:07.724620 21587 ts_manager.cc:194] Registered new tserver with Master: 711512a9a63e438c9ffbda240305be04 (127.20.53.67:37853)
I20250815 19:58:07.726058 21587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:43247
I20250815 19:58:07.726207 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:07.762279 21587 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48672:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 19:58:07.781432 21587 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 19:58:07.841068 21843 tablet_service.cc:1468] Processing CreateTablet for tablet afa52ab51f2446289f8bed67c02df4ea (DEFAULT_TABLE table=TestTable [id=d782781000244d17a459e2a096884dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:07.841988 21976 tablet_service.cc:1468] Processing CreateTablet for tablet afa52ab51f2446289f8bed67c02df4ea (DEFAULT_TABLE table=TestTable [id=d782781000244d17a459e2a096884dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:07.843024 21843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa52ab51f2446289f8bed67c02df4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:07.843629 21976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa52ab51f2446289f8bed67c02df4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:07.845026 21710 tablet_service.cc:1468] Processing CreateTablet for tablet afa52ab51f2446289f8bed67c02df4ea (DEFAULT_TABLE table=TestTable [id=d782781000244d17a459e2a096884dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:07.846983 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa52ab51f2446289f8bed67c02df4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:07.868348 22060 tablet_bootstrap.cc:492] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: Bootstrap starting.
I20250815 19:58:07.871668 22061 tablet_bootstrap.cc:492] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: Bootstrap starting.
I20250815 19:58:07.875752 22062 tablet_bootstrap.cc:492] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: Bootstrap starting.
I20250815 19:58:07.876436 22060 tablet_bootstrap.cc:654] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:07.879333 22061 tablet_bootstrap.cc:654] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:07.880340 22060 log.cc:826] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:07.881788 22061 log.cc:826] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:07.882558 22062 tablet_bootstrap.cc:654] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:07.884852 22062 log.cc:826] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:07.886169 22060 tablet_bootstrap.cc:492] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: No bootstrap required, opened a new log
I20250815 19:58:07.886682 22060 ts_tablet_manager.cc:1397] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20250815 19:58:07.888373 22061 tablet_bootstrap.cc:492] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: No bootstrap required, opened a new log
I20250815 19:58:07.888926 22061 ts_tablet_manager.cc:1397] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
I20250815 19:58:07.890693 22062 tablet_bootstrap.cc:492] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: No bootstrap required, opened a new log
I20250815 19:58:07.891081 22062 ts_tablet_manager.cc:1397] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20250815 19:58:07.915565 22060 raft_consensus.cc:357] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:07.916806 22060 raft_consensus.cc:738] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711512a9a63e438c9ffbda240305be04, State: Initialized, Role: FOLLOWER
I20250815 19:58:07.916640 22061 raft_consensus.cc:357] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:07.917640 22061 raft_consensus.cc:738] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e9fcc45b244962b2347a78e4c82b3d, State: Initialized, Role: FOLLOWER
I20250815 19:58:07.917594 22060 consensus_queue.cc:260] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:07.917812 22062 raft_consensus.cc:357] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:07.918807 22062 raft_consensus.cc:738] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4aad0f9f0648d793b41eba69e8a6ee, State: Initialized, Role: FOLLOWER
I20250815 19:58:07.918359 22061 consensus_queue.cc:260] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:07.919585 22062 consensus_queue.cc:260] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:07.926352 22061 ts_tablet_manager.cc:1428] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: Time spent starting tablet: real 0.037s user 0.030s sys 0.005s
I20250815 19:58:07.942021 22041 heartbeater.cc:499] Master 127.20.53.126:42149 was elected leader, sending a full tablet report...
I20250815 19:58:07.944058 22062 ts_tablet_manager.cc:1428] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: Time spent starting tablet: real 0.053s user 0.030s sys 0.022s
I20250815 19:58:07.946727 22060 ts_tablet_manager.cc:1428] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: Time spent starting tablet: real 0.060s user 0.030s sys 0.021s
W20250815 19:58:07.953632 22042 tablet.cc:2378] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:58:07.967065 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:07.970517 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1c4aad0f9f0648d793b41eba69e8a6ee to finish bootstrapping
I20250815 19:58:07.983942 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 76e9fcc45b244962b2347a78e4c82b3d to finish bootstrapping
I20250815 19:58:07.997838 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 711512a9a63e438c9ffbda240305be04 to finish bootstrapping
I20250815 19:58:08.039835 21730 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea"
dest_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
from {username='slave'} at 127.0.0.1:45844
I20250815 19:58:08.040383 21730 raft_consensus.cc:491] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 19:58:08.040684 21730 raft_consensus.cc:3058] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:08.045032 21730 raft_consensus.cc:513] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:08.047487 21730 leader_election.cc:290] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [CANDIDATE]: Term 1 election: Requested vote from peers 711512a9a63e438c9ffbda240305be04 (127.20.53.67:37853), 76e9fcc45b244962b2347a78e4c82b3d (127.20.53.66:41475)
I20250815 19:58:08.055750 20693 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250815 19:58:08.061851 21996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea" candidate_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "711512a9a63e438c9ffbda240305be04"
I20250815 19:58:08.061851 21863 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea" candidate_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76e9fcc45b244962b2347a78e4c82b3d"
I20250815 19:58:08.062575 21996 raft_consensus.cc:3058] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:08.062574 21863 raft_consensus.cc:3058] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:08.069393 21996 raft_consensus.cc:2466] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c4aad0f9f0648d793b41eba69e8a6ee in term 1.
I20250815 19:58:08.069393 21863 raft_consensus.cc:2466] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c4aad0f9f0648d793b41eba69e8a6ee in term 1.
I20250815 19:58:08.070688 21665 leader_election.cc:304] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c4aad0f9f0648d793b41eba69e8a6ee, 76e9fcc45b244962b2347a78e4c82b3d; no voters:
I20250815 19:58:08.071522 22068 raft_consensus.cc:2802] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:08.073235 22068 raft_consensus.cc:695] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 LEADER]: Becoming Leader. State: Replica: 1c4aad0f9f0648d793b41eba69e8a6ee, State: Running, Role: LEADER
I20250815 19:58:08.074043 22068 consensus_queue.cc:237] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:08.084504 21585 catalog_manager.cc:5582] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c4aad0f9f0648d793b41eba69e8a6ee (127.20.53.65). New cstate: current_term: 1 leader_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } health_report { overall_health: HEALTHY } } }
W20250815 19:58:08.129926 21776 tablet.cc:2378] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 19:58:08.156852 21909 tablet.cc:2378] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:58:08.161303 20693 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 19:58:08.366750 20693 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 19:58:08.525099 22068 consensus_queue.cc:1035] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 19:58:08.539858 22068 consensus_queue.cc:1035] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 19:58:10.351411 21730 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea"
dest_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45854
I20250815 19:58:10.352136 21730 raft_consensus.cc:604] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 LEADER]: Received request to transfer leadership
I20250815 19:58:10.515664 22089 raft_consensus.cc:991] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee: : Instructing follower 711512a9a63e438c9ffbda240305be04 to start an election
I20250815 19:58:10.516036 22089 raft_consensus.cc:1079] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 LEADER]: Signalling peer 711512a9a63e438c9ffbda240305be04 to start an election
I20250815 19:58:10.517417 21996 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea"
dest_uuid: "711512a9a63e438c9ffbda240305be04"
from {username='slave'} at 127.20.53.65:52541
I20250815 19:58:10.517884 21996 raft_consensus.cc:491] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 19:58:10.518107 21996 raft_consensus.cc:3058] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:58:10.522105 21996 raft_consensus.cc:513] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:10.524029 21996 leader_election.cc:290] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [CANDIDATE]: Term 2 election: Requested vote from peers 76e9fcc45b244962b2347a78e4c82b3d (127.20.53.66:41475), 1c4aad0f9f0648d793b41eba69e8a6ee (127.20.53.65:41249)
I20250815 19:58:10.534019 21863 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea" candidate_uuid: "711512a9a63e438c9ffbda240305be04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76e9fcc45b244962b2347a78e4c82b3d"
I20250815 19:58:10.534554 21863 raft_consensus.cc:3058] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:58:10.534617 21730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea" candidate_uuid: "711512a9a63e438c9ffbda240305be04" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
I20250815 19:58:10.535096 21730 raft_consensus.cc:3053] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 LEADER]: Stepping down as leader of term 1
I20250815 19:58:10.535317 21730 raft_consensus.cc:738] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c4aad0f9f0648d793b41eba69e8a6ee, State: Running, Role: LEADER
I20250815 19:58:10.535795 21730 consensus_queue.cc:260] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:10.536827 21730 raft_consensus.cc:3058] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:58:10.538936 21863 raft_consensus.cc:2466] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 711512a9a63e438c9ffbda240305be04 in term 2.
I20250815 19:58:10.539811 21931 leader_election.cc:304] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 711512a9a63e438c9ffbda240305be04, 76e9fcc45b244962b2347a78e4c82b3d; no voters:
I20250815 19:58:10.541533 21730 raft_consensus.cc:2466] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 711512a9a63e438c9ffbda240305be04 in term 2.
I20250815 19:58:10.541666 22110 raft_consensus.cc:2802] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 19:58:10.543390 22110 raft_consensus.cc:695] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [term 2 LEADER]: Becoming Leader. State: Replica: 711512a9a63e438c9ffbda240305be04, State: Running, Role: LEADER
I20250815 19:58:10.544186 22110 consensus_queue.cc:237] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } }
I20250815 19:58:10.551501 21586 catalog_manager.cc:5582] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 reported cstate change: term changed from 1 to 2, leader changed from 1c4aad0f9f0648d793b41eba69e8a6ee (127.20.53.65) to 711512a9a63e438c9ffbda240305be04 (127.20.53.67). New cstate: current_term: 2 leader_uuid: "711512a9a63e438c9ffbda240305be04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711512a9a63e438c9ffbda240305be04" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 } health_report { overall_health: UNKNOWN } } }
I20250815 19:58:10.977633 21863 raft_consensus.cc:1273] T afa52ab51f2446289f8bed67c02df4ea P 76e9fcc45b244962b2347a78e4c82b3d [term 2 FOLLOWER]: Refusing update from remote peer 711512a9a63e438c9ffbda240305be04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 19:58:10.978816 22110 consensus_queue.cc:1035] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e9fcc45b244962b2347a78e4c82b3d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250815 19:58:10.992187 21730 raft_consensus.cc:1273] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 2 FOLLOWER]: Refusing update from remote peer 711512a9a63e438c9ffbda240305be04: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 19:58:10.994196 22111 consensus_queue.cc:1035] T afa52ab51f2446289f8bed67c02df4ea P 711512a9a63e438c9ffbda240305be04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 41249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250815 19:58:13.095357 21730 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "afa52ab51f2446289f8bed67c02df4ea"
dest_uuid: "1c4aad0f9f0648d793b41eba69e8a6ee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45856
I20250815 19:58:13.096217 21730 raft_consensus.cc:604] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 2 FOLLOWER]: Received request to transfer leadership
I20250815 19:58:13.096673 21730 raft_consensus.cc:612] T afa52ab51f2446289f8bed67c02df4ea P 1c4aad0f9f0648d793b41eba69e8a6ee [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250815 19:58:14.134284 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21646
I20250815 19:58:14.158507 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21779
I20250815 19:58:14.184072 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21912
I20250815 19:58:14.209285 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 21554
2025-08-15T19:58:14Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14287 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250815 19:58:14.265415 20693 test_util.cc:276] Using random seed: -815100373
I20250815 19:58:14.269467 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 19:58:14.269630 20693 ts_itest-base.cc:116] --------------
I20250815 19:58:14.269783 20693 ts_itest-base.cc:117] 3 tablet servers
I20250815 19:58:14.269910 20693 ts_itest-base.cc:118] 3 replicas per TS
I20250815 19:58:14.270040 20693 ts_itest-base.cc:119] --------------
2025-08-15T19:58:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T19:58:14Z Disabled control of system clock
I20250815 19:58:14.306802 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:32925
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:32925 with env {}
W20250815 19:58:14.605643 22151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:14.606258 22151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:14.606673 22151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:14.637573 22151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:58:14.637880 22151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:14.638098 22151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:58:14.638302 22151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:58:14.675714 22151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:32925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:32925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:14.677078 22151 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:14.678694 22151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:14.689882 22157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:14.690454 22158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:15.857965 22159 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 19:58:15.859539 22160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:15.859567 22151 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:15.862569 22151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:15.865089 22151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:15.866427 22151 hybrid_clock.cc:648] HybridClock initialized: now 1755287895866382 us; error 50 us; skew 500 ppm
I20250815 19:58:15.867215 22151 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:15.873153 22151 webserver.cc:480] Webserver started at http://127.20.53.126:32915/ using document root <none> and password file <none>
I20250815 19:58:15.874050 22151 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:15.874257 22151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:15.874719 22151 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:15.879041 22151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0e0ed29ca2544c72b1294cbcd27fda94"
format_stamp: "Formatted at 2025-08-15 19:58:15 on dist-test-slave-d4bs"
I20250815 19:58:15.880092 22151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0e0ed29ca2544c72b1294cbcd27fda94"
format_stamp: "Formatted at 2025-08-15 19:58:15 on dist-test-slave-d4bs"
I20250815 19:58:15.887176 22151 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250815 19:58:15.892578 22167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:15.893638 22151 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250815 19:58:15.893993 22151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0e0ed29ca2544c72b1294cbcd27fda94"
format_stamp: "Formatted at 2025-08-15 19:58:15 on dist-test-slave-d4bs"
I20250815 19:58:15.894330 22151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:15.946478 22151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:15.947978 22151 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:15.948426 22151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:16.017361 22151 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:32925
I20250815 19:58:16.017417 22218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:32925 every 8 connection(s)
I20250815 19:58:16.020089 22151 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:58:16.024360 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22151
I20250815 19:58:16.024753 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 19:58:16.025318 22219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:16.051590 22219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94: Bootstrap starting.
I20250815 19:58:16.057080 22219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:16.058844 22219 log.cc:826] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:16.063189 22219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94: No bootstrap required, opened a new log
I20250815 19:58:16.080202 22219 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 32925 } }
I20250815 19:58:16.080869 22219 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:16.081073 22219 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0ed29ca2544c72b1294cbcd27fda94, State: Initialized, Role: FOLLOWER
I20250815 19:58:16.081684 22219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 32925 } }
I20250815 19:58:16.082139 22219 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:16.082386 22219 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:16.082656 22219 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:16.086588 22219 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 32925 } }
I20250815 19:58:16.087427 22219 leader_election.cc:304] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e0ed29ca2544c72b1294cbcd27fda94; no voters:
I20250815 19:58:16.089102 22219 leader_election.cc:290] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:16.089718 22224 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:16.091984 22224 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [term 1 LEADER]: Becoming Leader. State: Replica: 0e0ed29ca2544c72b1294cbcd27fda94, State: Running, Role: LEADER
I20250815 19:58:16.092847 22224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 32925 } }
I20250815 19:58:16.093791 22219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:58:16.104269 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 32925 } } }
I20250815 19:58:16.104159 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e0ed29ca2544c72b1294cbcd27fda94. Latest consensus state: current_term: 1 leader_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0ed29ca2544c72b1294cbcd27fda94" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 32925 } } }
I20250815 19:58:16.104871 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:16.105060 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94 [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:16.108263 22233 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:58:16.121003 22233 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:58:16.138759 22233 catalog_manager.cc:1349] Generated new cluster ID: 2159e8e54a3a49b7ad9915e8f6324699
I20250815 19:58:16.139034 22233 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:58:16.171061 22233 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 19:58:16.172472 22233 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:58:16.184403 22233 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0e0ed29ca2544c72b1294cbcd27fda94: Generated new TSK 0
I20250815 19:58:16.185252 22233 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 19:58:16.204162 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:32925
--builtin_ntp_servers=127.20.53.84:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 19:58:16.510124 22243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:16.510612 22243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:16.511107 22243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:16.541759 22243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:16.542651 22243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:58:16.577457 22243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:32925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:16.578755 22243 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:16.580318 22243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:16.592188 22249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:16.593988 22250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:16.602830 22252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:17.758443 22251 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 19:58:17.758520 22243 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:17.762805 22243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:17.765479 22243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:17.766959 22243 hybrid_clock.cc:648] HybridClock initialized: now 1755287897766895 us; error 81 us; skew 500 ppm
I20250815 19:58:17.767822 22243 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:17.774916 22243 webserver.cc:480] Webserver started at http://127.20.53.65:37723/ using document root <none> and password file <none>
I20250815 19:58:17.775883 22243 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:17.776113 22243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:17.776623 22243 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:17.781457 22243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "19f1f70ff1b34c82a37cb217d4d408a6"
format_stamp: "Formatted at 2025-08-15 19:58:17 on dist-test-slave-d4bs"
I20250815 19:58:17.782639 22243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "19f1f70ff1b34c82a37cb217d4d408a6"
format_stamp: "Formatted at 2025-08-15 19:58:17 on dist-test-slave-d4bs"
I20250815 19:58:17.790205 22243 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250815 19:58:17.796358 22259 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:17.797475 22243 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250815 19:58:17.797784 22243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "19f1f70ff1b34c82a37cb217d4d408a6"
format_stamp: "Formatted at 2025-08-15 19:58:17 on dist-test-slave-d4bs"
I20250815 19:58:17.798110 22243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:17.868988 22243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:17.870469 22243 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:17.870908 22243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:17.873481 22243 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:17.877535 22243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:17.877746 22243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:17.877992 22243 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:17.878156 22243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:18.008883 22243 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:40187
I20250815 19:58:18.008982 22371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:40187 every 8 connection(s)
I20250815 19:58:18.011456 22243 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:58:18.016431 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22243
I20250815 19:58:18.016997 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 19:58:18.026939 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:32925
--builtin_ntp_servers=127.20.53.84:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:58:18.049465 22372 heartbeater.cc:344] Connected to a master server at 127.20.53.126:32925
I20250815 19:58:18.049887 22372 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:18.050935 22372 heartbeater.cc:507] Master 127.20.53.126:32925 requested a full tablet report, sending...
I20250815 19:58:18.053290 22184 ts_manager.cc:194] Registered new tserver with Master: 19f1f70ff1b34c82a37cb217d4d408a6 (127.20.53.65:40187)
I20250815 19:58:18.055085 22184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:44965
W20250815 19:58:18.329252 22376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:18.329713 22376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:18.330132 22376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:18.361214 22376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:18.361995 22376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:58:18.395478 22376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:32925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:18.396665 22376 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:18.398168 22376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:18.409662 22382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:19.058260 22372 heartbeater.cc:499] Master 127.20.53.126:32925 was elected leader, sending a full tablet report...
W20250815 19:58:18.411103 22383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:18.413290 22385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:19.561656 22384 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 19:58:19.561795 22376 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:19.565619 22376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:19.568310 22376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:19.569728 22376 hybrid_clock.cc:648] HybridClock initialized: now 1755287899569694 us; error 59 us; skew 500 ppm
I20250815 19:58:19.570506 22376 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:19.576426 22376 webserver.cc:480] Webserver started at http://127.20.53.66:44043/ using document root <none> and password file <none>
I20250815 19:58:19.577401 22376 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:19.577605 22376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:19.578099 22376 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:19.582424 22376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bbfc5886accb4e83a567cb0cf333b88b"
format_stamp: "Formatted at 2025-08-15 19:58:19 on dist-test-slave-d4bs"
I20250815 19:58:19.583489 22376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bbfc5886accb4e83a567cb0cf333b88b"
format_stamp: "Formatted at 2025-08-15 19:58:19 on dist-test-slave-d4bs"
I20250815 19:58:19.590297 22376 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250815 19:58:19.595877 22392 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:19.596833 22376 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250815 19:58:19.597157 22376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bbfc5886accb4e83a567cb0cf333b88b"
format_stamp: "Formatted at 2025-08-15 19:58:19 on dist-test-slave-d4bs"
I20250815 19:58:19.597498 22376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:19.644831 22376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:19.646255 22376 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:19.646690 22376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:19.649274 22376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:19.655128 22376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:19.655365 22376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:19.655658 22376 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:19.655799 22376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:19.787626 22376 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:40401
I20250815 19:58:19.787731 22504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:40401 every 8 connection(s)
I20250815 19:58:19.790174 22376 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:58:19.791987 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22376
I20250815 19:58:19.792565 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 19:58:19.803710 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:32925
--builtin_ntp_servers=127.20.53.84:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:58:19.813015 22505 heartbeater.cc:344] Connected to a master server at 127.20.53.126:32925
I20250815 19:58:19.813439 22505 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:19.814440 22505 heartbeater.cc:507] Master 127.20.53.126:32925 requested a full tablet report, sending...
I20250815 19:58:19.816692 22184 ts_manager.cc:194] Registered new tserver with Master: bbfc5886accb4e83a567cb0cf333b88b (127.20.53.66:40401)
I20250815 19:58:19.818622 22184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:52423
W20250815 19:58:20.104903 22510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:20.105374 22510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:20.105924 22510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:20.137337 22510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:20.138151 22510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:58:20.171880 22510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:32925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:20.173143 22510 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:20.174636 22510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:20.186388 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:20.821772 22505 heartbeater.cc:499] Master 127.20.53.126:32925 was elected leader, sending a full tablet report...
W20250815 19:58:20.186874 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:21.460614 22518 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1272 milliseconds
W20250815 19:58:21.462205 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:21.467254 22510 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.281s user 0.003s sys 0.004s
W20250815 19:58:21.467617 22510 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.282s user 0.003s sys 0.004s
I20250815 19:58:21.467906 22510 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:21.469478 22510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:21.472304 22510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:21.473814 22510 hybrid_clock.cc:648] HybridClock initialized: now 1755287901473753 us; error 54 us; skew 500 ppm
I20250815 19:58:21.475001 22510 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:21.484202 22510 webserver.cc:480] Webserver started at http://127.20.53.67:38933/ using document root <none> and password file <none>
I20250815 19:58:21.485596 22510 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:21.485886 22510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:21.486547 22510 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:21.493896 22510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b3209364029f4084b875fc34f4a42b3a"
format_stamp: "Formatted at 2025-08-15 19:58:21 on dist-test-slave-d4bs"
I20250815 19:58:21.495522 22510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b3209364029f4084b875fc34f4a42b3a"
format_stamp: "Formatted at 2025-08-15 19:58:21 on dist-test-slave-d4bs"
I20250815 19:58:21.505641 22510 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20250815 19:58:21.513563 22526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:21.514816 22510 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250815 19:58:21.515231 22510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b3209364029f4084b875fc34f4a42b3a"
format_stamp: "Formatted at 2025-08-15 19:58:21 on dist-test-slave-d4bs"
I20250815 19:58:21.515735 22510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:21.589680 22510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:21.591154 22510 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:21.591630 22510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:21.594069 22510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:21.598053 22510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:21.598273 22510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:21.598513 22510 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:21.598670 22510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:21.736831 22510 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:37369
I20250815 19:58:21.736958 22638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:37369 every 8 connection(s)
I20250815 19:58:21.739344 22510 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:58:21.745234 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22510
I20250815 19:58:21.745618 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 19:58:21.760571 22639 heartbeater.cc:344] Connected to a master server at 127.20.53.126:32925
I20250815 19:58:21.761013 22639 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:21.762020 22639 heartbeater.cc:507] Master 127.20.53.126:32925 requested a full tablet report, sending...
I20250815 19:58:21.764286 22184 ts_manager.cc:194] Registered new tserver with Master: b3209364029f4084b875fc34f4a42b3a (127.20.53.67:37369)
I20250815 19:58:21.765591 22184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:51545
I20250815 19:58:21.778533 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:21.810850 22184 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49488:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 19:58:21.830189 22184 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 19:58:21.889322 22307 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd0b1b47f094847a4400075f0adcf10 (DEFAULT_TABLE table=TestTable [id=c3a68c347c784f08bdc9c6f3f7b2578e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:21.891346 22307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd0b1b47f094847a4400075f0adcf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:21.896937 22440 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd0b1b47f094847a4400075f0adcf10 (DEFAULT_TABLE table=TestTable [id=c3a68c347c784f08bdc9c6f3f7b2578e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:21.898470 22440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd0b1b47f094847a4400075f0adcf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:21.900185 22574 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd0b1b47f094847a4400075f0adcf10 (DEFAULT_TABLE table=TestTable [id=c3a68c347c784f08bdc9c6f3f7b2578e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:21.902112 22574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd0b1b47f094847a4400075f0adcf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:21.918207 22658 tablet_bootstrap.cc:492] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: Bootstrap starting.
I20250815 19:58:21.920101 22659 tablet_bootstrap.cc:492] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: Bootstrap starting.
I20250815 19:58:21.923995 22658 tablet_bootstrap.cc:654] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:21.927095 22659 tablet_bootstrap.cc:654] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:21.927295 22658 log.cc:826] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:21.929806 22659 log.cc:826] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:21.930876 22660 tablet_bootstrap.cc:492] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: Bootstrap starting.
I20250815 19:58:21.934964 22658 tablet_bootstrap.cc:492] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: No bootstrap required, opened a new log
I20250815 19:58:21.935256 22659 tablet_bootstrap.cc:492] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: No bootstrap required, opened a new log
I20250815 19:58:21.935508 22658 ts_tablet_manager.cc:1397] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20250815 19:58:21.935700 22659 ts_tablet_manager.cc:1397] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250815 19:58:21.937081 22660 tablet_bootstrap.cc:654] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:21.938748 22660 log.cc:826] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:21.943838 22660 tablet_bootstrap.cc:492] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: No bootstrap required, opened a new log
I20250815 19:58:21.944401 22660 ts_tablet_manager.cc:1397] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20250815 19:58:21.960908 22660 raft_consensus.cc:357] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:21.961611 22660 raft_consensus.cc:383] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:21.961865 22660 raft_consensus.cc:738] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3209364029f4084b875fc34f4a42b3a, State: Initialized, Role: FOLLOWER
I20250815 19:58:21.961817 22658 raft_consensus.cc:357] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:21.961819 22659 raft_consensus.cc:357] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:21.962693 22658 raft_consensus.cc:383] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:21.962693 22659 raft_consensus.cc:383] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:21.962592 22660 consensus_queue.cc:260] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:21.963011 22659 raft_consensus.cc:738] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbfc5886accb4e83a567cb0cf333b88b, State: Initialized, Role: FOLLOWER
I20250815 19:58:21.963012 22658 raft_consensus.cc:738] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f1f70ff1b34c82a37cb217d4d408a6, State: Initialized, Role: FOLLOWER
I20250815 19:58:21.963863 22658 consensus_queue.cc:260] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:21.963917 22659 consensus_queue.cc:260] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:21.968245 22639 heartbeater.cc:499] Master 127.20.53.126:32925 was elected leader, sending a full tablet report...
I20250815 19:58:21.969707 22660 ts_tablet_manager.cc:1428] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: Time spent starting tablet: real 0.025s user 0.023s sys 0.001s
I20250815 19:58:21.970350 22659 ts_tablet_manager.cc:1428] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: Time spent starting tablet: real 0.034s user 0.035s sys 0.000s
I20250815 19:58:21.974229 22658 ts_tablet_manager.cc:1428] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: Time spent starting tablet: real 0.038s user 0.027s sys 0.008s
W20250815 19:58:21.993770 22640 tablet.cc:2378] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 19:58:22.021517 22373 tablet.cc:2378] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 19:58:22.047260 22506 tablet.cc:2378] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:58:22.094456 22665 raft_consensus.cc:491] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:58:22.095023 22665 raft_consensus.cc:513] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:22.097519 22665 leader_election.cc:290] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f1f70ff1b34c82a37cb217d4d408a6 (127.20.53.65:40187), b3209364029f4084b875fc34f4a42b3a (127.20.53.67:37369)
I20250815 19:58:22.109510 22327 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd0b1b47f094847a4400075f0adcf10" candidate_uuid: "bbfc5886accb4e83a567cb0cf333b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" is_pre_election: true
I20250815 19:58:22.110342 22594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd0b1b47f094847a4400075f0adcf10" candidate_uuid: "bbfc5886accb4e83a567cb0cf333b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3209364029f4084b875fc34f4a42b3a" is_pre_election: true
I20250815 19:58:22.110446 22327 raft_consensus.cc:2466] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbfc5886accb4e83a567cb0cf333b88b in term 0.
I20250815 19:58:22.110991 22594 raft_consensus.cc:2466] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbfc5886accb4e83a567cb0cf333b88b in term 0.
I20250815 19:58:22.111598 22396 leader_election.cc:304] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f1f70ff1b34c82a37cb217d4d408a6, bbfc5886accb4e83a567cb0cf333b88b; no voters:
I20250815 19:58:22.112402 22665 raft_consensus.cc:2802] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 19:58:22.112668 22665 raft_consensus.cc:491] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:58:22.112982 22665 raft_consensus.cc:3058] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:22.117241 22665 raft_consensus.cc:513] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:22.118505 22665 leader_election.cc:290] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [CANDIDATE]: Term 1 election: Requested vote from peers 19f1f70ff1b34c82a37cb217d4d408a6 (127.20.53.65:40187), b3209364029f4084b875fc34f4a42b3a (127.20.53.67:37369)
I20250815 19:58:22.119352 22327 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd0b1b47f094847a4400075f0adcf10" candidate_uuid: "bbfc5886accb4e83a567cb0cf333b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f1f70ff1b34c82a37cb217d4d408a6"
I20250815 19:58:22.119411 22594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fd0b1b47f094847a4400075f0adcf10" candidate_uuid: "bbfc5886accb4e83a567cb0cf333b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3209364029f4084b875fc34f4a42b3a"
I20250815 19:58:22.119760 22327 raft_consensus.cc:3058] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:22.119817 22594 raft_consensus.cc:3058] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:22.124076 22327 raft_consensus.cc:2466] T 0fd0b1b47f094847a4400075f0adcf10 P 19f1f70ff1b34c82a37cb217d4d408a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbfc5886accb4e83a567cb0cf333b88b in term 1.
I20250815 19:58:22.124065 22594 raft_consensus.cc:2466] T 0fd0b1b47f094847a4400075f0adcf10 P b3209364029f4084b875fc34f4a42b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbfc5886accb4e83a567cb0cf333b88b in term 1.
I20250815 19:58:22.124949 22396 leader_election.cc:304] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3209364029f4084b875fc34f4a42b3a, bbfc5886accb4e83a567cb0cf333b88b; no voters:
I20250815 19:58:22.125522 22665 raft_consensus.cc:2802] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:22.127240 22665 raft_consensus.cc:695] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [term 1 LEADER]: Becoming Leader. State: Replica: bbfc5886accb4e83a567cb0cf333b88b, State: Running, Role: LEADER
I20250815 19:58:22.128037 22665 consensus_queue.cc:237] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } }
I20250815 19:58:22.138728 22184 catalog_manager.cc:5582] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b reported cstate change: term changed from 0 to 1, leader changed from <none> to bbfc5886accb4e83a567cb0cf333b88b (127.20.53.66). New cstate: current_term: 1 leader_uuid: "bbfc5886accb4e83a567cb0cf333b88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } health_report { overall_health: HEALTHY } } }
I20250815 19:58:22.174166 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:22.177386 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 19f1f70ff1b34c82a37cb217d4d408a6 to finish bootstrapping
I20250815 19:58:22.195444 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bbfc5886accb4e83a567cb0cf333b88b to finish bootstrapping
I20250815 19:58:22.206609 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b3209364029f4084b875fc34f4a42b3a to finish bootstrapping
I20250815 19:58:22.219945 22184 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49488:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250815 19:58:22.221730 22184 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 19:58:22.238255 22307 tablet_service.cc:1468] Processing CreateTablet for tablet 2c5b2e8933804d5f994f2101939ae48d (DEFAULT_TABLE table=TestAnotherTable [id=cf0e09cca1fd4e30b68c55cf06e16027]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 19:58:22.238862 22440 tablet_service.cc:1468] Processing CreateTablet for tablet 2c5b2e8933804d5f994f2101939ae48d (DEFAULT_TABLE table=TestAnotherTable [id=cf0e09cca1fd4e30b68c55cf06e16027]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 19:58:22.239434 22307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5b2e8933804d5f994f2101939ae48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:22.239451 22574 tablet_service.cc:1468] Processing CreateTablet for tablet 2c5b2e8933804d5f994f2101939ae48d (DEFAULT_TABLE table=TestAnotherTable [id=cf0e09cca1fd4e30b68c55cf06e16027]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 19:58:22.239921 22440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5b2e8933804d5f994f2101939ae48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:22.240507 22574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5b2e8933804d5f994f2101939ae48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:22.251257 22658 tablet_bootstrap.cc:492] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6: Bootstrap starting.
I20250815 19:58:22.254220 22659 tablet_bootstrap.cc:492] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b: Bootstrap starting.
I20250815 19:58:22.254477 22660 tablet_bootstrap.cc:492] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a: Bootstrap starting.
I20250815 19:58:22.256844 22658 tablet_bootstrap.cc:654] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:22.259512 22660 tablet_bootstrap.cc:654] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:22.261086 22659 tablet_bootstrap.cc:654] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:22.262969 22658 tablet_bootstrap.cc:492] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6: No bootstrap required, opened a new log
I20250815 19:58:22.263388 22658 ts_tablet_manager.cc:1397] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250815 19:58:22.264976 22660 tablet_bootstrap.cc:492] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a: No bootstrap required, opened a new log
I20250815 19:58:22.265363 22660 ts_tablet_manager.cc:1397] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20250815 19:58:22.266709 22658 raft_consensus.cc:357] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.267443 22660 raft_consensus.cc:357] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.267639 22658 raft_consensus.cc:383] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:22.267959 22660 raft_consensus.cc:383] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:22.267971 22658 raft_consensus.cc:738] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f1f70ff1b34c82a37cb217d4d408a6, State: Initialized, Role: FOLLOWER
I20250815 19:58:22.268175 22660 raft_consensus.cc:738] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3209364029f4084b875fc34f4a42b3a, State: Initialized, Role: FOLLOWER
I20250815 19:58:22.268921 22660 consensus_queue.cc:260] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.268803 22658 consensus_queue.cc:260] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.270795 22660 ts_tablet_manager.cc:1428] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250815 19:58:22.273329 22658 ts_tablet_manager.cc:1428] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250815 19:58:22.278934 22659 tablet_bootstrap.cc:492] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b: No bootstrap required, opened a new log
I20250815 19:58:22.279379 22659 ts_tablet_manager.cc:1397] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.014s
I20250815 19:58:22.282049 22659 raft_consensus.cc:357] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.282753 22659 raft_consensus.cc:383] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:22.283030 22659 raft_consensus.cc:738] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbfc5886accb4e83a567cb0cf333b88b, State: Initialized, Role: FOLLOWER
I20250815 19:58:22.283838 22659 consensus_queue.cc:260] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.292886 22659 ts_tablet_manager.cc:1428] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20250815 19:58:22.383468 22664 raft_consensus.cc:491] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:58:22.383872 22664 raft_consensus.cc:513] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.387950 22664 leader_election.cc:290] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f1f70ff1b34c82a37cb217d4d408a6 (127.20.53.65:40187), bbfc5886accb4e83a567cb0cf333b88b (127.20.53.66:40401)
I20250815 19:58:22.397020 22327 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5b2e8933804d5f994f2101939ae48d" candidate_uuid: "b3209364029f4084b875fc34f4a42b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" is_pre_election: true
I20250815 19:58:22.397524 22327 raft_consensus.cc:2466] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3209364029f4084b875fc34f4a42b3a in term 0.
I20250815 19:58:22.398185 22460 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5b2e8933804d5f994f2101939ae48d" candidate_uuid: "b3209364029f4084b875fc34f4a42b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbfc5886accb4e83a567cb0cf333b88b" is_pre_election: true
I20250815 19:58:22.398545 22530 leader_election.cc:304] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f1f70ff1b34c82a37cb217d4d408a6, b3209364029f4084b875fc34f4a42b3a; no voters:
I20250815 19:58:22.398770 22460 raft_consensus.cc:2466] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3209364029f4084b875fc34f4a42b3a in term 0.
I20250815 19:58:22.399238 22664 raft_consensus.cc:2802] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 19:58:22.399482 22664 raft_consensus.cc:491] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:58:22.399715 22664 raft_consensus.cc:3058] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:22.403882 22664 raft_consensus.cc:513] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.405339 22664 leader_election.cc:290] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [CANDIDATE]: Term 1 election: Requested vote from peers 19f1f70ff1b34c82a37cb217d4d408a6 (127.20.53.65:40187), bbfc5886accb4e83a567cb0cf333b88b (127.20.53.66:40401)
I20250815 19:58:22.405994 22327 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5b2e8933804d5f994f2101939ae48d" candidate_uuid: "b3209364029f4084b875fc34f4a42b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f1f70ff1b34c82a37cb217d4d408a6"
I20250815 19:58:22.406288 22460 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5b2e8933804d5f994f2101939ae48d" candidate_uuid: "b3209364029f4084b875fc34f4a42b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbfc5886accb4e83a567cb0cf333b88b"
I20250815 19:58:22.406533 22327 raft_consensus.cc:3058] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:22.406790 22460 raft_consensus.cc:3058] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:22.413076 22327 raft_consensus.cc:2466] T 2c5b2e8933804d5f994f2101939ae48d P 19f1f70ff1b34c82a37cb217d4d408a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3209364029f4084b875fc34f4a42b3a in term 1.
I20250815 19:58:22.413348 22460 raft_consensus.cc:2466] T 2c5b2e8933804d5f994f2101939ae48d P bbfc5886accb4e83a567cb0cf333b88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3209364029f4084b875fc34f4a42b3a in term 1.
I20250815 19:58:22.414044 22530 leader_election.cc:304] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f1f70ff1b34c82a37cb217d4d408a6, b3209364029f4084b875fc34f4a42b3a; no voters:
I20250815 19:58:22.414646 22664 raft_consensus.cc:2802] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:22.416165 22664 raft_consensus.cc:695] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [term 1 LEADER]: Becoming Leader. State: Replica: b3209364029f4084b875fc34f4a42b3a, State: Running, Role: LEADER
I20250815 19:58:22.417352 22664 consensus_queue.cc:237] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } }
I20250815 19:58:22.426146 22184 catalog_manager.cc:5582] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to b3209364029f4084b875fc34f4a42b3a (127.20.53.67). New cstate: current_term: 1 leader_uuid: "b3209364029f4084b875fc34f4a42b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 } health_report { overall_health: HEALTHY } } }
I20250815 19:58:22.516459 22665 consensus_queue.cc:1035] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3209364029f4084b875fc34f4a42b3a" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 19:58:22.532405 22665 consensus_queue.cc:1035] T 0fd0b1b47f094847a4400075f0adcf10 P bbfc5886accb4e83a567cb0cf333b88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 19:58:22.762919 22679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:22.763517 22679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:22.795351 22679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 19:58:22.801983 22664 consensus_queue.cc:1035] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbfc5886accb4e83a567cb0cf333b88b" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 19:58:22.816891 22664 consensus_queue.cc:1035] T 2c5b2e8933804d5f994f2101939ae48d P b3209364029f4084b875fc34f4a42b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f1f70ff1b34c82a37cb217d4d408a6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 19:58:24.148083 22679 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.303s user 0.498s sys 0.801s
W20250815 19:58:24.148375 22679 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.303s user 0.498s sys 0.801s
W20250815 19:58:25.509395 22706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:25.509991 22706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:25.541105 22706 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 19:58:26.839962 22706 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.260s user 0.492s sys 0.765s
W20250815 19:58:26.840262 22706 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.260s user 0.492s sys 0.765s
W20250815 19:58:28.215708 22720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:28.216276 22720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:28.247603 22720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 19:58:29.522651 22720 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.236s user 0.405s sys 0.828s
W20250815 19:58:29.522956 22720 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.236s user 0.405s sys 0.828s
W20250815 19:58:30.915163 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:30.915771 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:30.946789 22736 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 19:58:32.242554 22736 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.257s user 0.486s sys 0.769s
W20250815 19:58:32.242868 22736 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.257s user 0.486s sys 0.769s
I20250815 19:58:33.310374 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22243
I20250815 19:58:33.332041 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22376
I20250815 19:58:33.356961 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22510
I20250815 19:58:33.383278 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22151
2025-08-15T19:58:33Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19176 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250815 19:58:33.442644 20693 test_util.cc:276] Using random seed: -795923143
I20250815 19:58:33.446779 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 19:58:33.446954 20693 ts_itest-base.cc:116] --------------
I20250815 19:58:33.447106 20693 ts_itest-base.cc:117] 3 tablet servers
I20250815 19:58:33.447247 20693 ts_itest-base.cc:118] 3 replicas per TS
I20250815 19:58:33.447386 20693 ts_itest-base.cc:119] --------------
2025-08-15T19:58:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T19:58:33Z Disabled control of system clock
I20250815 19:58:33.483951 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:39783
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:39783
--superuser_acl=no-such-user with env {}
W20250815 19:58:33.781572 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:33.782105 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:33.782546 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:33.813331 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:58:33.813627 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:33.813825 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:58:33.814013 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:58:33.848767 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:39783
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:39783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:33.849972 22757 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:33.851588 22757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:33.862427 22763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:35.267424 22762 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 22757
W20250815 19:58:35.658457 22762 kernel_stack_watchdog.cc:198] Thread 22757 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:58:33.862983 22764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:35.661479 22757 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.800s user 0.712s sys 1.087s
W20250815 19:58:35.661964 22757 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.800s user 0.712s sys 1.087s
W20250815 19:58:35.664039 22766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:35.669170 22765 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1801 milliseconds
I20250815 19:58:35.669260 22757 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:35.671025 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:35.673969 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:35.675421 22757 hybrid_clock.cc:648] HybridClock initialized: now 1755287915675346 us; error 82 us; skew 500 ppm
I20250815 19:58:35.676550 22757 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:35.683912 22757 webserver.cc:480] Webserver started at http://127.20.53.126:34209/ using document root <none> and password file <none>
I20250815 19:58:35.685258 22757 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:35.685633 22757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:35.686242 22757 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:35.690631 22757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6d2c1bef1b46423d87fd472cda0238ec"
format_stamp: "Formatted at 2025-08-15 19:58:35 on dist-test-slave-d4bs"
I20250815 19:58:35.691689 22757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6d2c1bef1b46423d87fd472cda0238ec"
format_stamp: "Formatted at 2025-08-15 19:58:35 on dist-test-slave-d4bs"
I20250815 19:58:35.698736 22757 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 19:58:35.703851 22773 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:35.704849 22757 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250815 19:58:35.705154 22757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6d2c1bef1b46423d87fd472cda0238ec"
format_stamp: "Formatted at 2025-08-15 19:58:35 on dist-test-slave-d4bs"
I20250815 19:58:35.705490 22757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:35.755625 22757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:35.757647 22757 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:35.758118 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:35.827569 22757 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:39783
I20250815 19:58:35.827680 22824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:39783 every 8 connection(s)
I20250815 19:58:35.830262 22757 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:58:35.831419 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22757
I20250815 19:58:35.831854 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 19:58:35.837404 22825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:35.861390 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec: Bootstrap starting.
I20250815 19:58:35.866808 22825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:35.868511 22825 log.cc:826] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:35.873231 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec: No bootstrap required, opened a new log
I20250815 19:58:35.890224 22825 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c1bef1b46423d87fd472cda0238ec" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39783 } }
I20250815 19:58:35.890910 22825 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:35.891162 22825 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2c1bef1b46423d87fd472cda0238ec, State: Initialized, Role: FOLLOWER
I20250815 19:58:35.891924 22825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c1bef1b46423d87fd472cda0238ec" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39783 } }
I20250815 19:58:35.892437 22825 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:35.892741 22825 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:35.893080 22825 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:35.897239 22825 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c1bef1b46423d87fd472cda0238ec" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39783 } }
I20250815 19:58:35.897992 22825 leader_election.cc:304] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d2c1bef1b46423d87fd472cda0238ec; no voters:
I20250815 19:58:35.899650 22825 leader_election.cc:290] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:35.900393 22830 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:35.902485 22830 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [term 1 LEADER]: Becoming Leader. State: Replica: 6d2c1bef1b46423d87fd472cda0238ec, State: Running, Role: LEADER
I20250815 19:58:35.903209 22830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c1bef1b46423d87fd472cda0238ec" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39783 } }
I20250815 19:58:35.904227 22825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:58:35.911609 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d2c1bef1b46423d87fd472cda0238ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c1bef1b46423d87fd472cda0238ec" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39783 } } }
I20250815 19:58:35.912407 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:35.912395 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d2c1bef1b46423d87fd472cda0238ec. Latest consensus state: current_term: 1 leader_uuid: "6d2c1bef1b46423d87fd472cda0238ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2c1bef1b46423d87fd472cda0238ec" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39783 } } }
I20250815 19:58:35.913408 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:35.916934 22837 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:58:35.929419 22837 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:58:35.947716 22837 catalog_manager.cc:1349] Generated new cluster ID: 9167e25d06df4a0e9b4dd350e2b77f61
I20250815 19:58:35.947965 22837 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:58:35.982800 22837 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 19:58:35.984869 22837 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:58:36.007334 22837 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6d2c1bef1b46423d87fd472cda0238ec: Generated new TSK 0
I20250815 19:58:36.008222 22837 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 19:58:36.020104 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39783
--builtin_ntp_servers=127.20.53.84:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 19:58:36.352355 22849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:36.352876 22849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:36.353389 22849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:36.384043 22849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:36.385002 22849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:58:36.419117 22849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39783
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:36.420464 22849 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:36.422119 22849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:36.435439 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:36.437065 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:37.620757 22857 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 19:58:37.622692 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:37.626358 22849 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.191s user 0.484s sys 0.702s
W20250815 19:58:37.626722 22849 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.192s user 0.484s sys 0.702s
I20250815 19:58:37.627007 22849 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:37.628434 22849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:37.631382 22849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:37.632849 22849 hybrid_clock.cc:648] HybridClock initialized: now 1755287917632791 us; error 59 us; skew 500 ppm
I20250815 19:58:37.634055 22849 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:37.642658 22849 webserver.cc:480] Webserver started at http://127.20.53.65:37503/ using document root <none> and password file <none>
I20250815 19:58:37.643829 22849 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:37.644078 22849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:37.644598 22849 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:37.652267 22849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5bfd14959ff347999bc9f8db14bb1676"
format_stamp: "Formatted at 2025-08-15 19:58:37 on dist-test-slave-d4bs"
I20250815 19:58:37.653784 22849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5bfd14959ff347999bc9f8db14bb1676"
format_stamp: "Formatted at 2025-08-15 19:58:37 on dist-test-slave-d4bs"
I20250815 19:58:37.662775 22849 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.006s
I20250815 19:58:37.670157 22865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:37.671403 22849 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250815 19:58:37.671783 22849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5bfd14959ff347999bc9f8db14bb1676"
format_stamp: "Formatted at 2025-08-15 19:58:37 on dist-test-slave-d4bs"
I20250815 19:58:37.672192 22849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:37.752197 22849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:37.754097 22849 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:37.754633 22849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:37.757738 22849 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:37.763216 22849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:37.763484 22849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:37.763756 22849 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:37.763942 22849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:37.926236 22849 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:46613
I20250815 19:58:37.926335 22977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:46613 every 8 connection(s)
I20250815 19:58:37.928948 22849 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:58:37.932379 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22849
I20250815 19:58:37.932967 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 19:58:37.943964 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39783
--builtin_ntp_servers=127.20.53.84:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:58:37.978196 22978 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39783
I20250815 19:58:37.978708 22978 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:37.979974 22978 heartbeater.cc:507] Master 127.20.53.126:39783 requested a full tablet report, sending...
I20250815 19:58:37.983098 22790 ts_manager.cc:194] Registered new tserver with Master: 5bfd14959ff347999bc9f8db14bb1676 (127.20.53.65:46613)
I20250815 19:58:37.985914 22790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:44623
W20250815 19:58:38.257957 22982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:38.258472 22982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:38.258977 22982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:38.290809 22982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:38.291671 22982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:58:38.326520 22982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39783
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:38.327901 22982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:38.329537 22982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:38.342986 22988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:38.990059 22978 heartbeater.cc:499] Master 127.20.53.126:39783 was elected leader, sending a full tablet report...
W20250815 19:58:38.344096 22989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:39.743299 22991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:39.745347 22982 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.402s user 0.478s sys 0.891s
W20250815 19:58:39.745627 22982 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.403s user 0.478s sys 0.891s
W20250815 19:58:39.745532 22990 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1401 milliseconds
I20250815 19:58:39.745877 22982 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:39.747023 22982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:39.749315 22982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:39.750699 22982 hybrid_clock.cc:648] HybridClock initialized: now 1755287919750667 us; error 44 us; skew 500 ppm
I20250815 19:58:39.751538 22982 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:39.758692 22982 webserver.cc:480] Webserver started at http://127.20.53.66:39899/ using document root <none> and password file <none>
I20250815 19:58:39.759718 22982 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:39.759963 22982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:39.760509 22982 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:39.765095 22982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "501567a91f724e6a9b7ccbdf4683a16d"
format_stamp: "Formatted at 2025-08-15 19:58:39 on dist-test-slave-d4bs"
I20250815 19:58:39.766196 22982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "501567a91f724e6a9b7ccbdf4683a16d"
format_stamp: "Formatted at 2025-08-15 19:58:39 on dist-test-slave-d4bs"
I20250815 19:58:39.774257 22982 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 19:58:39.782402 22998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:39.783809 22982 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20250815 19:58:39.784256 22982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "501567a91f724e6a9b7ccbdf4683a16d"
format_stamp: "Formatted at 2025-08-15 19:58:39 on dist-test-slave-d4bs"
I20250815 19:58:39.784749 22982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:39.881805 22982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:39.883232 22982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:39.883613 22982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:39.886080 22982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:39.890008 22982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:39.890210 22982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:39.890403 22982 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:39.890534 22982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:40.021706 22982 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:42305
I20250815 19:58:40.021803 23111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:42305 every 8 connection(s)
I20250815 19:58:40.024298 22982 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:58:40.033519 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 22982
I20250815 19:58:40.034168 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 19:58:40.041841 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39783
--builtin_ntp_servers=127.20.53.84:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:58:40.048349 23112 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39783
I20250815 19:58:40.048841 23112 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:40.050007 23112 heartbeater.cc:507] Master 127.20.53.126:39783 requested a full tablet report, sending...
I20250815 19:58:40.052505 22790 ts_manager.cc:194] Registered new tserver with Master: 501567a91f724e6a9b7ccbdf4683a16d (127.20.53.66:42305)
I20250815 19:58:40.053869 22790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:47907
W20250815 19:58:40.337188 23116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:40.337694 23116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:40.338212 23116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:40.369031 23116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:40.369894 23116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:58:40.403766 23116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39783
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:40.405148 23116 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:40.406764 23116 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:40.419207 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:40.430928 22974 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac4c0 after lost signal to thread 22850
W20250815 19:58:40.432147 22974 debug-util.cc:398] Leaking SignalData structure 0x7b08000b68a0 after lost signal to thread 22977
I20250815 19:58:41.057375 23112 heartbeater.cc:499] Master 127.20.53.126:39783 was elected leader, sending a full tablet report...
W20250815 19:58:41.821434 23121 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23116
W20250815 19:58:42.074924 23116 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.656s user 0.529s sys 1.116s
W20250815 19:58:42.075793 23116 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.657s user 0.529s sys 1.117s
W20250815 19:58:40.419488 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:42.076118 23124 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1655 milliseconds
I20250815 19:58:42.077195 23116 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 19:58:42.077286 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:42.080821 23116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:42.083006 23116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:42.084347 23116 hybrid_clock.cc:648] HybridClock initialized: now 1755287922084326 us; error 46 us; skew 500 ppm
I20250815 19:58:42.085171 23116 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:42.092072 23116 webserver.cc:480] Webserver started at http://127.20.53.67:46451/ using document root <none> and password file <none>
I20250815 19:58:42.093055 23116 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:42.093287 23116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:42.093744 23116 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:42.098572 23116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1f28cb82a21e4e54ae8dd2738101d2c6"
format_stamp: "Formatted at 2025-08-15 19:58:42 on dist-test-slave-d4bs"
I20250815 19:58:42.099663 23116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1f28cb82a21e4e54ae8dd2738101d2c6"
format_stamp: "Formatted at 2025-08-15 19:58:42 on dist-test-slave-d4bs"
I20250815 19:58:42.106887 23116 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250815 19:58:42.112704 23132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:42.113852 23116 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 19:58:42.114193 23116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1f28cb82a21e4e54ae8dd2738101d2c6"
format_stamp: "Formatted at 2025-08-15 19:58:42 on dist-test-slave-d4bs"
I20250815 19:58:42.114527 23116 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:42.177968 23116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:42.179431 23116 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:42.179854 23116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:42.182920 23116 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:42.187176 23116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:42.187393 23116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:42.187640 23116 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:42.187809 23116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:42.359453 23116 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:35373
I20250815 19:58:42.359550 23244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:35373 every 8 connection(s)
I20250815 19:58:42.362174 23116 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:58:42.364588 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 23116
I20250815 19:58:42.365064 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 19:58:42.398573 23245 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39783
I20250815 19:58:42.399139 23245 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:42.400622 23245 heartbeater.cc:507] Master 127.20.53.126:39783 requested a full tablet report, sending...
I20250815 19:58:42.403539 22790 ts_manager.cc:194] Registered new tserver with Master: 1f28cb82a21e4e54ae8dd2738101d2c6 (127.20.53.67:35373)
I20250815 19:58:42.405409 22790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:46959
I20250815 19:58:42.408218 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:42.451737 22790 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 19:58:42.483625 22790 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 19:58:42.552649 23047 tablet_service.cc:1468] Processing CreateTablet for tablet 9f44bfe031324ec88303e794e50d1407 (DEFAULT_TABLE table=TestTable [id=6d1bb35c0c394748a9ab264c87bf3b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:42.552866 23180 tablet_service.cc:1468] Processing CreateTablet for tablet 9f44bfe031324ec88303e794e50d1407 (DEFAULT_TABLE table=TestTable [id=6d1bb35c0c394748a9ab264c87bf3b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:42.554117 22913 tablet_service.cc:1468] Processing CreateTablet for tablet 9f44bfe031324ec88303e794e50d1407 (DEFAULT_TABLE table=TestTable [id=6d1bb35c0c394748a9ab264c87bf3b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:42.554826 23180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f44bfe031324ec88303e794e50d1407. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:42.554865 23047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f44bfe031324ec88303e794e50d1407. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:42.555984 22913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f44bfe031324ec88303e794e50d1407. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:42.587605 23265 tablet_bootstrap.cc:492] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: Bootstrap starting.
I20250815 19:58:42.596266 23264 tablet_bootstrap.cc:492] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: Bootstrap starting.
I20250815 19:58:42.597164 23266 tablet_bootstrap.cc:492] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: Bootstrap starting.
I20250815 19:58:42.604393 23264 tablet_bootstrap.cc:654] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:42.604570 23265 tablet_bootstrap.cc:654] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:42.604629 23266 tablet_bootstrap.cc:654] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:42.607012 23265 log.cc:826] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:42.607024 23266 log.cc:826] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:42.607185 23264 log.cc:826] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:42.615185 23264 tablet_bootstrap.cc:492] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: No bootstrap required, opened a new log
I20250815 19:58:42.615935 23264 ts_tablet_manager.cc:1397] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20250815 19:58:42.627146 23266 tablet_bootstrap.cc:492] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: No bootstrap required, opened a new log
I20250815 19:58:42.627149 23265 tablet_bootstrap.cc:492] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: No bootstrap required, opened a new log
I20250815 19:58:42.627643 23266 ts_tablet_manager.cc:1397] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.022s
I20250815 19:58:42.627643 23265 ts_tablet_manager.cc:1397] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.018s
I20250815 19:58:42.654397 23265 raft_consensus.cc:357] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.654395 23266 raft_consensus.cc:357] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.655320 23266 raft_consensus.cc:383] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:42.655530 23265 raft_consensus.cc:383] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:42.655840 23265 raft_consensus.cc:738] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f28cb82a21e4e54ae8dd2738101d2c6, State: Initialized, Role: FOLLOWER
I20250815 19:58:42.655650 23266 raft_consensus.cc:738] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501567a91f724e6a9b7ccbdf4683a16d, State: Initialized, Role: FOLLOWER
I20250815 19:58:42.656332 23264 raft_consensus.cc:357] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.656772 23265 consensus_queue.cc:260] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.656769 23266 consensus_queue.cc:260] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.657385 23264 raft_consensus.cc:383] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:42.657770 23264 raft_consensus.cc:738] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bfd14959ff347999bc9f8db14bb1676, State: Initialized, Role: FOLLOWER
I20250815 19:58:42.658849 23264 consensus_queue.cc:260] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.666877 23265 ts_tablet_manager.cc:1428] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: Time spent starting tablet: real 0.039s user 0.017s sys 0.021s
I20250815 19:58:42.667872 23245 heartbeater.cc:499] Master 127.20.53.126:39783 was elected leader, sending a full tablet report...
I20250815 19:58:42.683780 23266 ts_tablet_manager.cc:1428] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: Time spent starting tablet: real 0.056s user 0.023s sys 0.030s
I20250815 19:58:42.690307 23264 ts_tablet_manager.cc:1428] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: Time spent starting tablet: real 0.074s user 0.037s sys 0.022s
W20250815 19:58:42.715348 22979 tablet.cc:2378] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:58:42.757386 23270 raft_consensus.cc:491] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:58:42.758034 23270 raft_consensus.cc:513] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.764938 23270 leader_election.cc:290] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bfd14959ff347999bc9f8db14bb1676 (127.20.53.65:46613), 501567a91f724e6a9b7ccbdf4683a16d (127.20.53.66:42305)
I20250815 19:58:42.770716 22933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f44bfe031324ec88303e794e50d1407" candidate_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bfd14959ff347999bc9f8db14bb1676" is_pre_election: true
I20250815 19:58:42.771651 22933 raft_consensus.cc:2466] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f28cb82a21e4e54ae8dd2738101d2c6 in term 0.
I20250815 19:58:42.772953 23133 leader_election.cc:304] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f28cb82a21e4e54ae8dd2738101d2c6, 5bfd14959ff347999bc9f8db14bb1676; no voters:
I20250815 19:58:42.773885 23270 raft_consensus.cc:2802] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 19:58:42.774216 23270 raft_consensus.cc:491] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:58:42.775152 23270 raft_consensus.cc:3058] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:42.775652 23067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f44bfe031324ec88303e794e50d1407" candidate_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501567a91f724e6a9b7ccbdf4683a16d" is_pre_election: true
I20250815 19:58:42.776654 23067 raft_consensus.cc:2466] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f28cb82a21e4e54ae8dd2738101d2c6 in term 0.
I20250815 19:58:42.780110 23270 raft_consensus.cc:513] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.781491 23270 leader_election.cc:290] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [CANDIDATE]: Term 1 election: Requested vote from peers 5bfd14959ff347999bc9f8db14bb1676 (127.20.53.65:46613), 501567a91f724e6a9b7ccbdf4683a16d (127.20.53.66:42305)
I20250815 19:58:42.782423 22933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f44bfe031324ec88303e794e50d1407" candidate_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bfd14959ff347999bc9f8db14bb1676"
I20250815 19:58:42.782569 23067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f44bfe031324ec88303e794e50d1407" candidate_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501567a91f724e6a9b7ccbdf4683a16d"
I20250815 19:58:42.782857 22933 raft_consensus.cc:3058] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:42.783118 23067 raft_consensus.cc:3058] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [term 0 FOLLOWER]: Advancing to term 1
W20250815 19:58:42.785295 23113 tablet.cc:2378] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:58:42.788733 22933 raft_consensus.cc:2466] T 9f44bfe031324ec88303e794e50d1407 P 5bfd14959ff347999bc9f8db14bb1676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f28cb82a21e4e54ae8dd2738101d2c6 in term 1.
I20250815 19:58:42.789717 23133 leader_election.cc:304] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f28cb82a21e4e54ae8dd2738101d2c6, 5bfd14959ff347999bc9f8db14bb1676; no voters:
I20250815 19:58:42.790485 23067 raft_consensus.cc:2466] T 9f44bfe031324ec88303e794e50d1407 P 501567a91f724e6a9b7ccbdf4683a16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f28cb82a21e4e54ae8dd2738101d2c6 in term 1.
I20250815 19:58:42.790565 23270 raft_consensus.cc:2802] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:42.804827 23270 raft_consensus.cc:695] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [term 1 LEADER]: Becoming Leader. State: Replica: 1f28cb82a21e4e54ae8dd2738101d2c6, State: Running, Role: LEADER
I20250815 19:58:42.805661 23270 consensus_queue.cc:237] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } }
I20250815 19:58:42.818033 22789 catalog_manager.cc:5582] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f28cb82a21e4e54ae8dd2738101d2c6 (127.20.53.67). New cstate: current_term: 1 leader_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f28cb82a21e4e54ae8dd2738101d2c6" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250815 19:58:42.865654 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:42.870082 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5bfd14959ff347999bc9f8db14bb1676 to finish bootstrapping
W20250815 19:58:42.880034 23246 tablet.cc:2378] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:58:42.887709 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 501567a91f724e6a9b7ccbdf4683a16d to finish bootstrapping
I20250815 19:58:42.902611 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1f28cb82a21e4e54ae8dd2738101d2c6 to finish bootstrapping
I20250815 19:58:43.386869 23288 consensus_queue.cc:1035] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bfd14959ff347999bc9f8db14bb1676" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 46613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 19:58:43.395139 23288 consensus_queue.cc:1035] T 9f44bfe031324ec88303e794e50d1407 P 1f28cb82a21e4e54ae8dd2738101d2c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501567a91f724e6a9b7ccbdf4683a16d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 19:58:44.625308 22789 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:52634
I20250815 19:58:45.655887 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22849
I20250815 19:58:45.683635 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22982
I20250815 19:58:45.711354 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 23116
I20250815 19:58:45.735882 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 22757
2025-08-15T19:58:45Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (12344 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250815 19:58:45.787381 20693 test_util.cc:276] Using random seed: -783578404
I20250815 19:58:45.791507 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 19:58:45.791658 20693 ts_itest-base.cc:116] --------------
I20250815 19:58:45.791774 20693 ts_itest-base.cc:117] 3 tablet servers
I20250815 19:58:45.791883 20693 ts_itest-base.cc:118] 3 replicas per TS
I20250815 19:58:45.791986 20693 ts_itest-base.cc:119] --------------
2025-08-15T19:58:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T19:58:45Z Disabled control of system clock
I20250815 19:58:45.831586 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:34801
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:34801 with env {}
W20250815 19:58:46.124176 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:46.124783 23313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:46.125211 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:46.154800 23313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:58:46.155102 23313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:46.155359 23313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:58:46.155591 23313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:58:46.189702 23313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:34801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:46.191011 23313 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:46.192920 23313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:46.202869 23319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:46.204993 23320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:47.390257 23322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:47.392346 23321 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1183 milliseconds
I20250815 19:58:47.392457 23313 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:47.393689 23313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:47.396685 23313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:47.398108 23313 hybrid_clock.cc:648] HybridClock initialized: now 1755287927398065 us; error 50 us; skew 500 ppm
I20250815 19:58:47.398911 23313 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:47.405365 23313 webserver.cc:480] Webserver started at http://127.20.53.126:42575/ using document root <none> and password file <none>
I20250815 19:58:47.406306 23313 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:47.406514 23313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:47.406970 23313 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:47.411321 23313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e50e1ff9f73d437eaf425225b75437c2"
format_stamp: "Formatted at 2025-08-15 19:58:47 on dist-test-slave-d4bs"
I20250815 19:58:47.412401 23313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e50e1ff9f73d437eaf425225b75437c2"
format_stamp: "Formatted at 2025-08-15 19:58:47 on dist-test-slave-d4bs"
I20250815 19:58:47.419445 23313 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250815 19:58:47.424847 23329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:47.425837 23313 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250815 19:58:47.426151 23313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e50e1ff9f73d437eaf425225b75437c2"
format_stamp: "Formatted at 2025-08-15 19:58:47 on dist-test-slave-d4bs"
I20250815 19:58:47.426492 23313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:47.472843 23313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:47.474166 23313 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:47.474586 23313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:47.542560 23313 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:34801
I20250815 19:58:47.542629 23380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:34801 every 8 connection(s)
I20250815 19:58:47.545218 23313 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:58:47.549908 23381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:47.555382 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 23313
I20250815 19:58:47.555836 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 19:58:47.573253 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap starting.
I20250815 19:58:47.578228 23381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:47.580238 23381 log.cc:826] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:47.584486 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: No bootstrap required, opened a new log
I20250815 19:58:47.601459 23381 raft_consensus.cc:357] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:58:47.602046 23381 raft_consensus.cc:383] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:47.602237 23381 raft_consensus.cc:738] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Initialized, Role: FOLLOWER
I20250815 19:58:47.602793 23381 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:58:47.603250 23381 raft_consensus.cc:397] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:47.603457 23381 raft_consensus.cc:491] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:47.603706 23381 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:47.607528 23381 raft_consensus.cc:513] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:58:47.608090 23381 leader_election.cc:304] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e50e1ff9f73d437eaf425225b75437c2; no voters:
I20250815 19:58:47.609880 23381 leader_election.cc:290] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:47.610739 23386 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:47.612917 23386 raft_consensus.cc:695] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 1 LEADER]: Becoming Leader. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Running, Role: LEADER
I20250815 19:58:47.613591 23386 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:58:47.614454 23381 sys_catalog.cc:564] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:58:47.622994 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:58:47.623392 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50e1ff9f73d437eaf425225b75437c2. Latest consensus state: current_term: 1 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:58:47.623888 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:47.623888 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:58:47.629348 23394 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:58:47.640173 23394 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:58:47.658972 23394 catalog_manager.cc:1349] Generated new cluster ID: 2ebeb2378ff247dca7924065e794a7d6
I20250815 19:58:47.659291 23394 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:58:47.696524 23394 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 19:58:47.697973 23394 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:58:47.713996 23394 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Generated new TSK 0
I20250815 19:58:47.714907 23394 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 19:58:47.724197 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:34801
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 19:58:48.031494 23405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:48.032047 23405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:48.032552 23405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:48.063603 23405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:48.064512 23405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:58:48.099850 23405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:48.101269 23405 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:48.102788 23405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:48.114916 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:49.518211 23410 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23405
W20250815 19:58:49.582341 23410 kernel_stack_watchdog.cc:198] Thread 23405 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:58:48.116371 23412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:49.584093 23413 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1466 milliseconds
W20250815 19:58:49.584695 23405 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.468s user 0.451s sys 1.011s
W20250815 19:58:49.585197 23405 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.469s user 0.451s sys 1.011s
I20250815 19:58:49.585741 23405 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 19:58:49.585772 23414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:49.589103 23405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:49.591665 23405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:49.593106 23405 hybrid_clock.cc:648] HybridClock initialized: now 1755287929593070 us; error 52 us; skew 500 ppm
I20250815 19:58:49.593855 23405 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:49.600425 23405 webserver.cc:480] Webserver started at http://127.20.53.65:43373/ using document root <none> and password file <none>
I20250815 19:58:49.601341 23405 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:49.601570 23405 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:49.601980 23405 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:49.606233 23405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "992c23efe942442db73e5fbcef76c9f6"
format_stamp: "Formatted at 2025-08-15 19:58:49 on dist-test-slave-d4bs"
I20250815 19:58:49.607261 23405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "992c23efe942442db73e5fbcef76c9f6"
format_stamp: "Formatted at 2025-08-15 19:58:49 on dist-test-slave-d4bs"
I20250815 19:58:49.614952 23405 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.009s
I20250815 19:58:49.621302 23421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:49.622354 23405 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250815 19:58:49.622669 23405 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "992c23efe942442db73e5fbcef76c9f6"
format_stamp: "Formatted at 2025-08-15 19:58:49 on dist-test-slave-d4bs"
I20250815 19:58:49.622946 23405 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:49.687683 23405 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:49.689188 23405 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:49.689597 23405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:49.692193 23405 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:49.696741 23405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:49.696950 23405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 19:58:49.697213 23405 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:49.697366 23405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:49.858368 23405 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:40217
I20250815 19:58:49.858498 23533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:40217 every 8 connection(s)
I20250815 19:58:49.861258 23405 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:58:49.871268 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 23405
I20250815 19:58:49.871774 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 19:58:49.879689 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:34801
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:58:49.888499 23534 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:58:49.889048 23534 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:49.890489 23534 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:58:49.893749 23346 ts_manager.cc:194] Registered new tserver with Master: 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217)
I20250815 19:58:49.896566 23346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:51525
W20250815 19:58:50.183993 23539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:50.184514 23539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:50.185245 23539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:50.216367 23539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:50.217243 23539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:58:50.251463 23539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:50.252776 23539 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:50.254266 23539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:50.266479 23545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:50.900822 23534 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
W20250815 19:58:50.268034 23546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:51.453264 23547 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1183 milliseconds
W20250815 19:58:51.454600 23548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:51.456665 23539 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.189s user 0.410s sys 0.775s
W20250815 19:58:51.456971 23539 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.189s user 0.410s sys 0.779s
I20250815 19:58:51.457198 23539 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:51.459036 23539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:51.461310 23539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:51.462662 23539 hybrid_clock.cc:648] HybridClock initialized: now 1755287931462635 us; error 46 us; skew 500 ppm
I20250815 19:58:51.463442 23539 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:51.470638 23539 webserver.cc:480] Webserver started at http://127.20.53.66:35999/ using document root <none> and password file <none>
I20250815 19:58:51.471668 23539 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:51.471900 23539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:51.472333 23539 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:51.476615 23539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d93df400fe844250aef143a864c15a9d"
format_stamp: "Formatted at 2025-08-15 19:58:51 on dist-test-slave-d4bs"
I20250815 19:58:51.477717 23539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d93df400fe844250aef143a864c15a9d"
format_stamp: "Formatted at 2025-08-15 19:58:51 on dist-test-slave-d4bs"
I20250815 19:58:51.485136 23539 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250815 19:58:51.491189 23555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:51.492352 23539 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250815 19:58:51.492681 23539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d93df400fe844250aef143a864c15a9d"
format_stamp: "Formatted at 2025-08-15 19:58:51 on dist-test-slave-d4bs"
I20250815 19:58:51.493039 23539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:51.573606 23539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:51.575013 23539 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:51.575464 23539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:51.577956 23539 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:51.581976 23539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:51.582221 23539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:51.582477 23539 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:51.582646 23539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:51.714327 23539 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:35271
I20250815 19:58:51.714426 23667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:35271 every 8 connection(s)
I20250815 19:58:51.716928 23539 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:58:51.719020 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 23539
I20250815 19:58:51.719703 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 19:58:51.727319 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:34801
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:58:51.738711 23668 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:58:51.739140 23668 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:51.740134 23668 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:58:51.742246 23346 ts_manager.cc:194] Registered new tserver with Master: d93df400fe844250aef143a864c15a9d (127.20.53.66:35271)
I20250815 19:58:51.743510 23346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:36541
W20250815 19:58:52.033005 23672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:58:52.033516 23672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:58:52.033993 23672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:58:52.065351 23672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:58:52.066293 23672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:58:52.099509 23672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:58:52.100943 23672 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:58:52.102548 23672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:58:52.113850 23678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:58:52.746716 23668 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
W20250815 19:58:53.518430 23677 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23672
W20250815 19:58:53.710073 23677 kernel_stack_watchdog.cc:198] Thread 23672 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:58:52.114657 23679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:53.710803 23672 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.596s user 0.532s sys 1.012s
W20250815 19:58:53.711311 23672 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.597s user 0.533s sys 1.012s
W20250815 19:58:53.713371 23681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:58:53.717633 23680 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1602 milliseconds
I20250815 19:58:53.717669 23672 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:58:53.719126 23672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:58:53.721774 23672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:58:53.723273 23672 hybrid_clock.cc:648] HybridClock initialized: now 1755287933723209 us; error 47 us; skew 500 ppm
I20250815 19:58:53.724359 23672 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:58:53.731127 23672 webserver.cc:480] Webserver started at http://127.20.53.67:33979/ using document root <none> and password file <none>
I20250815 19:58:53.732081 23672 fs_manager.cc:362] Metadata directory not provided
I20250815 19:58:53.732291 23672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:58:53.732685 23672 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:58:53.737088 23672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d87be07713104bb7828f7250198f837b"
format_stamp: "Formatted at 2025-08-15 19:58:53 on dist-test-slave-d4bs"
I20250815 19:58:53.738095 23672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d87be07713104bb7828f7250198f837b"
format_stamp: "Formatted at 2025-08-15 19:58:53 on dist-test-slave-d4bs"
I20250815 19:58:53.745008 23672 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250815 19:58:53.750202 23688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:53.751189 23672 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250815 19:58:53.751514 23672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d87be07713104bb7828f7250198f837b"
format_stamp: "Formatted at 2025-08-15 19:58:53 on dist-test-slave-d4bs"
I20250815 19:58:53.751829 23672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:58:53.806452 23672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:58:53.807840 23672 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:58:53.808285 23672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:58:53.810760 23672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:58:53.814677 23672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:58:53.814882 23672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:58:53.815105 23672 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:58:53.815343 23672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250815 19:58:53.948626 23672 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:35047
I20250815 19:58:53.948755 23800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:35047 every 8 connection(s)
I20250815 19:58:53.951146 23672 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:58:53.961580 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 23672
I20250815 19:58:53.962075 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 19:58:53.977336 23801 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:58:53.977746 23801 heartbeater.cc:461] Registering TS with master...
I20250815 19:58:53.978832 23801 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:58:53.981118 23346 ts_manager.cc:194] Registered new tserver with Master: d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
I20250815 19:58:53.982395 23346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:39981
I20250815 19:58:53.995545 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:54.028889 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:58:54.029208 20693 test_util.cc:276] Using random seed: -775336576
I20250815 19:58:54.069262 23346 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37932:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 19:58:54.109654 23736 tablet_service.cc:1468] Processing CreateTablet for tablet 9644c93dde714878a568a83a6bbd8a9d (DEFAULT_TABLE table=TestTable [id=4144a2526203406cac6f036e03c0b03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:54.111166 23736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9644c93dde714878a568a83a6bbd8a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:54.129231 23821 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap starting.
I20250815 19:58:54.134636 23821 tablet_bootstrap.cc:654] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:54.136116 23821 log.cc:826] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:54.140460 23821 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: No bootstrap required, opened a new log
I20250815 19:58:54.140862 23821 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20250815 19:58:54.156934 23821 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:58:54.157408 23821 raft_consensus.cc:383] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:54.157615 23821 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87be07713104bb7828f7250198f837b, State: Initialized, Role: FOLLOWER
I20250815 19:58:54.158172 23821 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:58:54.158640 23821 raft_consensus.cc:397] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:54.158874 23821 raft_consensus.cc:491] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:54.159286 23821 raft_consensus.cc:3058] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:54.163420 23821 raft_consensus.cc:513] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:58:54.164081 23821 leader_election.cc:304] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d87be07713104bb7828f7250198f837b; no voters:
I20250815 19:58:54.165726 23821 leader_election.cc:290] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:54.166442 23823 raft_consensus.cc:2802] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:54.168445 23823 raft_consensus.cc:695] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 LEADER]: Becoming Leader. State: Replica: d87be07713104bb7828f7250198f837b, State: Running, Role: LEADER
I20250815 19:58:54.168799 23801 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
I20250815 19:58:54.169294 23823 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:58:54.169719 23821 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Time spent starting tablet: real 0.029s user 0.016s sys 0.012s
I20250815 19:58:54.180642 23346 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b reported cstate change: term changed from 0 to 1, leader changed from <none> to d87be07713104bb7828f7250198f837b (127.20.53.67). New cstate: current_term: 1 leader_uuid: "d87be07713104bb7828f7250198f837b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } health_report { overall_health: HEALTHY } } }
I20250815 19:58:54.388314 20693 test_util.cc:276] Using random seed: -774977476
I20250815 19:58:54.407363 23346 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55134:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 19:58:54.435032 23469 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3c262e14ae4134b7031e6a90066522 (DEFAULT_TABLE table=TestTable1 [id=5d1a3e4f07d74b2b8281c7b31e6a4a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:54.436462 23469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3c262e14ae4134b7031e6a90066522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:54.458429 23842 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap starting.
I20250815 19:58:54.463837 23842 tablet_bootstrap.cc:654] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:54.465586 23842 log.cc:826] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:54.470029 23842 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: No bootstrap required, opened a new log
I20250815 19:58:54.470420 23842 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250815 19:58:54.486999 23842 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:58:54.487525 23842 raft_consensus.cc:383] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:54.487737 23842 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Initialized, Role: FOLLOWER
I20250815 19:58:54.488358 23842 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:58:54.489060 23842 raft_consensus.cc:397] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:54.489392 23842 raft_consensus.cc:491] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:54.489732 23842 raft_consensus.cc:3058] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:54.494038 23842 raft_consensus.cc:513] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:58:54.494740 23842 leader_election.cc:304] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6; no voters:
I20250815 19:58:54.496487 23842 leader_election.cc:290] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:54.496879 23844 raft_consensus.cc:2802] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:54.499310 23844 raft_consensus.cc:695] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 LEADER]: Becoming Leader. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Running, Role: LEADER
I20250815 19:58:54.500535 23842 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Time spent starting tablet: real 0.030s user 0.026s sys 0.004s
I20250815 19:58:54.500202 23844 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:58:54.510882 23346 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65). New cstate: current_term: 1 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } health_report { overall_health: HEALTHY } } }
I20250815 19:58:54.681296 20693 test_util.cc:276] Using random seed: -774684504
I20250815 19:58:54.702356 23346 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55140:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 19:58:54.733403 23603 tablet_service.cc:1468] Processing CreateTablet for tablet ab2b302e74d541578e701d63a5bbe304 (DEFAULT_TABLE table=TestTable2 [id=192f2dcb486249c59cb348216a45965d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:58:54.734809 23603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2b302e74d541578e701d63a5bbe304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:58:54.754413 23863 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:58:54.760409 23863 tablet_bootstrap.cc:654] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Neither blocks nor log segments found. Creating new log.
I20250815 19:58:54.762228 23863 log.cc:826] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Log is configured to *not* fsync() on all Append() calls
I20250815 19:58:54.766783 23863 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: No bootstrap required, opened a new log
I20250815 19:58:54.767205 23863 ts_tablet_manager.cc:1397] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20250815 19:58:54.784781 23863 raft_consensus.cc:357] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:58:54.785362 23863 raft_consensus.cc:383] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:58:54.785632 23863 raft_consensus.cc:738] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: FOLLOWER
I20250815 19:58:54.786319 23863 consensus_queue.cc:260] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:58:54.786831 23863 raft_consensus.cc:397] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:58:54.787094 23863 raft_consensus.cc:491] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:58:54.787398 23863 raft_consensus.cc:3058] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:58:54.791842 23863 raft_consensus.cc:513] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:58:54.792523 23863 leader_election.cc:304] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d93df400fe844250aef143a864c15a9d; no voters:
I20250815 19:58:54.794257 23863 leader_election.cc:290] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:58:54.794625 23865 raft_consensus.cc:2802] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:58:54.797217 23865 raft_consensus.cc:695] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 LEADER]: Becoming Leader. State: Replica: d93df400fe844250aef143a864c15a9d, State: Running, Role: LEADER
I20250815 19:58:54.798341 23863 ts_tablet_manager.cc:1428] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.031s user 0.025s sys 0.007s
I20250815 19:58:54.798525 23865 consensus_queue.cc:237] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:58:54.807030 23346 catalog_manager.cc:5582] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d reported cstate change: term changed from 0 to 1, leader changed from <none> to d93df400fe844250aef143a864c15a9d (127.20.53.66). New cstate: current_term: 1 leader_uuid: "d93df400fe844250aef143a864c15a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } health_report { overall_health: HEALTHY } } }
I20250815 19:58:55.005653 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 23313
W20250815 19:58:55.195535 23801 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:34801 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:34801: connect: Connection refused (error 111)
W20250815 19:58:55.527994 23534 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:34801 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:34801: connect: Connection refused (error 111)
W20250815 19:58:55.838151 23668 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:34801 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:34801: connect: Connection refused (error 111)
I20250815 19:58:59.801910 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 23405
I20250815 19:58:59.825404 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 23539
I20250815 19:58:59.851791 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 23672
I20250815 19:58:59.877835 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:34801
--webserver_interface=127.20.53.126
--webserver_port=42575
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:34801 with env {}
W20250815 19:59:00.178874 23941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:00.179476 23941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:00.179910 23941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:00.210395 23941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:59:00.210729 23941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:00.210987 23941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:59:00.211254 23941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:59:00.245311 23941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:34801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=42575
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:00.246649 23941 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:00.248206 23941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:00.258715 23948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:00.259153 23949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:01.452116 23951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:01.453855 23950 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1189 milliseconds
W20250815 19:59:01.455281 23941 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.196s user 0.456s sys 0.738s
W20250815 19:59:01.455538 23941 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.197s user 0.456s sys 0.738s
I20250815 19:59:01.455783 23941 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:01.456923 23941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:01.459592 23941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:01.460938 23941 hybrid_clock.cc:648] HybridClock initialized: now 1755287941460893 us; error 64 us; skew 500 ppm
I20250815 19:59:01.461736 23941 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:01.469519 23941 webserver.cc:480] Webserver started at http://127.20.53.126:42575/ using document root <none> and password file <none>
I20250815 19:59:01.470422 23941 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:01.470647 23941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:01.478772 23941 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250815 19:59:01.483611 23958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:01.484812 23941 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250815 19:59:01.485167 23941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e50e1ff9f73d437eaf425225b75437c2"
format_stamp: "Formatted at 2025-08-15 19:58:47 on dist-test-slave-d4bs"
I20250815 19:59:01.487098 23941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:01.555878 23941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:01.557401 23941 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:01.557843 23941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:01.626801 23941 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:34801
I20250815 19:59:01.626852 24009 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:34801 every 8 connection(s)
I20250815 19:59:01.629573 23941 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:59:01.632083 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 23941
I20250815 19:59:01.634442 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:40217
--local_ip_for_outbound_sockets=127.20.53.65
--tserver_master_addrs=127.20.53.126:34801
--webserver_port=43373
--webserver_interface=127.20.53.65
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:59:01.644369 24010 sys_catalog.cc:263] Verifying existing consensus state
I20250815 19:59:01.648905 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap starting.
I20250815 19:59:01.662164 24010 log.cc:826] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:01.708954 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:01.709769 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap complete.
I20250815 19:59:01.730077 24010 raft_consensus.cc:357] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:01.732195 24010 raft_consensus.cc:738] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Initialized, Role: FOLLOWER
I20250815 19:59:01.733052 24010 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:01.733566 24010 raft_consensus.cc:397] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:01.733839 24010 raft_consensus.cc:491] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:01.734162 24010 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 2 FOLLOWER]: Advancing to term 3
I20250815 19:59:01.739904 24010 raft_consensus.cc:513] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:01.740568 24010 leader_election.cc:304] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e50e1ff9f73d437eaf425225b75437c2; no voters:
I20250815 19:59:01.742777 24010 leader_election.cc:290] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 19:59:01.743162 24014 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 19:59:01.746687 24014 raft_consensus.cc:695] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 LEADER]: Becoming Leader. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Running, Role: LEADER
I20250815 19:59:01.747570 24014 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:01.747931 24010 sys_catalog.cc:564] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:59:01.755316 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50e1ff9f73d437eaf425225b75437c2. Latest consensus state: current_term: 3 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:59:01.756662 24015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:59:01.757488 24015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:01.758685 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:01.768363 24021 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:59:01.782819 24021 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=192f2dcb486249c59cb348216a45965d]
I20250815 19:59:01.785029 24021 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=3bb3a8ea90fb455dbd6f289280f383dc]
I20250815 19:59:01.787046 24021 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d41405a80eb24aae86a9b93130743baa]
I20250815 19:59:01.796902 24021 tablet_loader.cc:96] loaded metadata for tablet 6e3c262e14ae4134b7031e6a90066522 (table TestTable1 [id=3bb3a8ea90fb455dbd6f289280f383dc])
I20250815 19:59:01.798571 24021 tablet_loader.cc:96] loaded metadata for tablet 9644c93dde714878a568a83a6bbd8a9d (table TestTable [id=d41405a80eb24aae86a9b93130743baa])
I20250815 19:59:01.800347 24021 tablet_loader.cc:96] loaded metadata for tablet ab2b302e74d541578e701d63a5bbe304 (table TestTable2 [id=192f2dcb486249c59cb348216a45965d])
I20250815 19:59:01.802311 24021 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:59:01.808826 24021 catalog_manager.cc:1261] Loaded cluster ID: 2ebeb2378ff247dca7924065e794a7d6
I20250815 19:59:01.809376 24021 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:59:01.818917 24021 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:59:01.819067 24031 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250815 19:59:01.822881 24021 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Loaded TSK: 0
I20250815 19:59:01.823932 24021 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 19:59:01.985446 24012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:01.985999 24012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:01.986507 24012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:02.018694 24012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:02.019593 24012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:59:02.054493 24012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:40217
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=43373
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:02.055795 24012 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:02.057391 24012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:02.069589 24037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:03.474323 24036 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24012
W20250815 19:59:03.514940 24012 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.445s user 0.469s sys 0.916s
W20250815 19:59:02.071868 24038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:03.515558 24012 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.446s user 0.469s sys 0.916s
W20250815 19:59:03.518186 24040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:03.520219 24039 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1449 milliseconds
I20250815 19:59:03.520246 24012 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:03.521525 24012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:03.523636 24012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:03.524971 24012 hybrid_clock.cc:648] HybridClock initialized: now 1755287943524920 us; error 55 us; skew 500 ppm
I20250815 19:59:03.525704 24012 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:03.532295 24012 webserver.cc:480] Webserver started at http://127.20.53.65:43373/ using document root <none> and password file <none>
I20250815 19:59:03.533233 24012 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:03.533460 24012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:03.541533 24012 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250815 19:59:03.546427 24047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:03.547585 24012 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250815 19:59:03.547919 24012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "992c23efe942442db73e5fbcef76c9f6"
format_stamp: "Formatted at 2025-08-15 19:58:49 on dist-test-slave-d4bs"
I20250815 19:59:03.549942 24012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:03.596874 24012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:03.598244 24012 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:03.598659 24012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:03.601625 24012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:03.608160 24054 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 19:59:03.615649 24012 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 19:59:03.615861 24012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250815 19:59:03.616135 24012 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 19:59:03.620548 24012 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 19:59:03.620819 24012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250815 19:59:03.621176 24054 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap starting.
I20250815 19:59:03.695716 24054 log.cc:826] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:03.804785 24054 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:03.805872 24054 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap complete.
I20250815 19:59:03.807829 24054 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Time spent bootstrapping tablet: real 0.187s user 0.142s sys 0.040s
I20250815 19:59:03.831221 24012 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:40217
I20250815 19:59:03.831342 24161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:40217 every 8 connection(s)
I20250815 19:59:03.829912 24054 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:59:03.832875 24054 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Initialized, Role: FOLLOWER
I20250815 19:59:03.834141 24012 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:59:03.833817 24054 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:59:03.834518 24054 raft_consensus.cc:397] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:03.834846 24054 raft_consensus.cc:491] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:03.835209 24054 raft_consensus.cc:3058] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:59:03.843420 24054 raft_consensus.cc:513] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:59:03.844542 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 24012
I20250815 19:59:03.844333 24054 leader_election.cc:304] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6; no voters:
I20250815 19:59:03.846623 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:35271
--local_ip_for_outbound_sockets=127.20.53.66
--tserver_master_addrs=127.20.53.126:34801
--webserver_port=35999
--webserver_interface=127.20.53.66
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:59:03.860652 24054 leader_election.cc:290] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 19:59:03.861584 24166 raft_consensus.cc:2802] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 19:59:03.873838 24166 raft_consensus.cc:695] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEADER]: Becoming Leader. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Running, Role: LEADER
I20250815 19:59:03.875142 24166 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } }
I20250815 19:59:03.889709 24054 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Time spent starting tablet: real 0.082s user 0.058s sys 0.021s
I20250815 19:59:03.896273 24162 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:59:03.896876 24162 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:03.898088 24162 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:59:03.909691 23975 ts_manager.cc:194] Registered new tserver with Master: 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217)
I20250815 19:59:03.913426 23975 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: term changed from 0 to 2, leader changed from <none> to 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65), VOTER 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) added. New cstate: current_term: 2 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } health_report { overall_health: HEALTHY } } }
W20250815 19:59:03.960049 23975 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e3c262e14ae4134b7031e6a90066522 with cas_config_opid_index -1: no extra replica candidate found for tablet 6e3c262e14ae4134b7031e6a90066522 (table TestTable1 [id=3bb3a8ea90fb455dbd6f289280f383dc]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250815 19:59:03.962396 23975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:42829
I20250815 19:59:03.966495 24162 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
W20250815 19:59:04.204514 24167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:04.205041 24167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:04.205526 24167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:04.236052 24167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:04.237136 24167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:59:04.271198 24167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:35271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=35999
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:04.272573 24167 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:04.274257 24167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:04.286526 24181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:04.291131 24184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:04.290362 24182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:05.494974 24183 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1203 milliseconds
I20250815 19:59:05.495095 24167 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:05.496344 24167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:05.498497 24167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:05.499817 24167 hybrid_clock.cc:648] HybridClock initialized: now 1755287945499796 us; error 46 us; skew 500 ppm
I20250815 19:59:05.500574 24167 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:05.506877 24167 webserver.cc:480] Webserver started at http://127.20.53.66:35999/ using document root <none> and password file <none>
I20250815 19:59:05.507752 24167 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:05.507949 24167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:05.515681 24167 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250815 19:59:05.520349 24191 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:05.521350 24167 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250815 19:59:05.521678 24167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d93df400fe844250aef143a864c15a9d"
format_stamp: "Formatted at 2025-08-15 19:58:51 on dist-test-slave-d4bs"
I20250815 19:59:05.523538 24167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:05.582113 24167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:05.583724 24167 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:05.584124 24167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:05.586679 24167 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:05.592417 24198 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 19:59:05.599897 24167 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 19:59:05.600142 24167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250815 19:59:05.600426 24167 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 19:59:05.605044 24167 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 19:59:05.605230 24167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250815 19:59:05.605563 24198 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:59:05.664817 24198 log.cc:826] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:05.746886 24198 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:05.747771 24198 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap complete.
I20250815 19:59:05.749229 24198 ts_tablet_manager.cc:1397] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.144s user 0.104s sys 0.037s
I20250815 19:59:05.766458 24198 raft_consensus.cc:357] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:05.769764 24198 raft_consensus.cc:738] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: FOLLOWER
I20250815 19:59:05.770617 24198 consensus_queue.cc:260] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:05.771139 24198 raft_consensus.cc:397] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:05.771575 24198 raft_consensus.cc:491] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:05.771996 24198 raft_consensus.cc:3058] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:59:05.778235 24198 raft_consensus.cc:513] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:05.778885 24198 leader_election.cc:304] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d93df400fe844250aef143a864c15a9d; no voters:
I20250815 19:59:05.781123 24198 leader_election.cc:290] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 19:59:05.781507 24298 raft_consensus.cc:2802] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Leader election won for term 2
I20250815 19:59:05.788482 24198 ts_tablet_manager.cc:1428] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.039s user 0.030s sys 0.004s
I20250815 19:59:05.789127 24298 raft_consensus.cc:695] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 LEADER]: Becoming Leader. State: Replica: d93df400fe844250aef143a864c15a9d, State: Running, Role: LEADER
I20250815 19:59:05.789886 24298 consensus_queue.cc:237] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:05.798473 24167 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:35271
I20250815 19:59:05.799178 24310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:35271 every 8 connection(s)
I20250815 19:59:05.801048 24167 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:59:05.802747 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 24167
I20250815 19:59:05.804932 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:35047
--local_ip_for_outbound_sockets=127.20.53.67
--tserver_master_addrs=127.20.53.126:34801
--webserver_port=33979
--webserver_interface=127.20.53.67
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:59:05.822530 24311 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:59:05.823069 24311 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:05.824375 24311 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:59:05.828912 23975 ts_manager.cc:194] Registered new tserver with Master: d93df400fe844250aef143a864c15a9d (127.20.53.66:35271)
I20250815 19:59:05.830232 23975 catalog_manager.cc:5582] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d93df400fe844250aef143a864c15a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } health_report { overall_health: HEALTHY } } }
I20250815 19:59:05.842762 23975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:52619
I20250815 19:59:05.846940 24311 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
W20250815 19:59:06.120746 24315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:06.121251 24315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:06.121687 24315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:06.152601 24315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:06.153438 24315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:59:06.187630 24315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:35047
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=33979
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:06.188860 24315 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:06.190467 24315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:06.202085 24325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:06.296175 24117 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:06.310967 24334 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } } }
I20250815 19:59:06.323906 23962 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e3c262e14ae4134b7031e6a90066522 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250815 19:59:06.330916 23975 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index -1 to 8, NON_VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) added. New cstate: current_term: 2 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 19:59:06.342171 24051 consensus_peers.cc:489] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 -> Peer d93df400fe844250aef143a864c15a9d (127.20.53.66:35271): Couldn't send request to peer d93df400fe844250aef143a864c15a9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e3c262e14ae4134b7031e6a90066522. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:06.883805 24339 ts_tablet_manager.cc:927] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Initiating tablet copy from peer 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217)
I20250815 19:59:06.913264 24339 tablet_copy_client.cc:323] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.53.65:40217
I20250815 19:59:06.946175 24137 tablet_copy_service.cc:140] P 992c23efe942442db73e5fbcef76c9f6: Received BeginTabletCopySession request for tablet 6e3c262e14ae4134b7031e6a90066522 from peer d93df400fe844250aef143a864c15a9d ({username='slave'} at 127.20.53.66:48437)
I20250815 19:59:06.946969 24137 tablet_copy_service.cc:161] P 992c23efe942442db73e5fbcef76c9f6: Beginning new tablet copy session on tablet 6e3c262e14ae4134b7031e6a90066522 from peer d93df400fe844250aef143a864c15a9d at {username='slave'} at 127.20.53.66:48437: session id = d93df400fe844250aef143a864c15a9d-6e3c262e14ae4134b7031e6a90066522
I20250815 19:59:06.956812 24137 tablet_copy_source_session.cc:215] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 19:59:06.962508 24339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3c262e14ae4134b7031e6a90066522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:06.984400 24339 tablet_copy_client.cc:806] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: tablet copy: Starting download of 0 data blocks...
I20250815 19:59:06.985195 24339 tablet_copy_client.cc:670] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: tablet copy: Starting download of 1 WAL segments...
I20250815 19:59:06.991667 24339 tablet_copy_client.cc:538] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 19:59:07.004215 24339 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:59:07.167090 24339 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:07.168262 24339 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Bootstrap complete.
I20250815 19:59:07.169142 24339 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.165s user 0.159s sys 0.008s
I20250815 19:59:07.172597 24339 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:07.173453 24339 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: LEARNER
I20250815 19:59:07.174242 24339 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:07.178923 24339 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250815 19:59:07.183988 24137 tablet_copy_service.cc:342] P 992c23efe942442db73e5fbcef76c9f6: Request end of tablet copy session d93df400fe844250aef143a864c15a9d-6e3c262e14ae4134b7031e6a90066522 received from {username='slave'} at 127.20.53.66:48437
I20250815 19:59:07.184626 24137 tablet_copy_service.cc:434] P 992c23efe942442db73e5fbcef76c9f6: ending tablet copy session d93df400fe844250aef143a864c15a9d-6e3c262e14ae4134b7031e6a90066522 on tablet 6e3c262e14ae4134b7031e6a90066522 with peer d93df400fe844250aef143a864c15a9d
I20250815 19:59:07.374346 24261 raft_consensus.cc:1215] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250815 19:59:06.202828 24326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:07.455178 24328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:07.457305 24327 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1251 milliseconds
I20250815 19:59:07.457437 24315 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:07.458688 24315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:07.460784 24315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:07.462122 24315 hybrid_clock.cc:648] HybridClock initialized: now 1755287947462083 us; error 48 us; skew 500 ppm
I20250815 19:59:07.463001 24315 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:07.469005 24315 webserver.cc:480] Webserver started at http://127.20.53.67:33979/ using document root <none> and password file <none>
I20250815 19:59:07.470147 24315 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:07.470391 24315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:07.478276 24315 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250815 19:59:07.482739 24351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:07.483772 24315 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 19:59:07.484146 24315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d87be07713104bb7828f7250198f837b"
format_stamp: "Formatted at 2025-08-15 19:58:53 on dist-test-slave-d4bs"
I20250815 19:59:07.486114 24315 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:07.542634 24315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:07.544095 24315 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:07.544536 24315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:07.546983 24315 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:07.552538 24358 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 19:59:07.560166 24315 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 19:59:07.560452 24315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250815 19:59:07.560752 24315 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 19:59:07.565172 24315 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 19:59:07.565382 24315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250815 19:59:07.565709 24358 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap starting.
I20250815 19:59:07.619437 24358 log.cc:826] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:07.719501 24358 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:07.720399 24358 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap complete.
I20250815 19:59:07.721735 24358 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Time spent bootstrapping tablet: real 0.156s user 0.104s sys 0.047s
I20250815 19:59:07.740878 24358 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:59:07.744071 24358 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87be07713104bb7828f7250198f837b, State: Initialized, Role: FOLLOWER
I20250815 19:59:07.744961 24358 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:59:07.745577 24358 raft_consensus.cc:397] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:07.745837 24358 raft_consensus.cc:491] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:07.746232 24358 raft_consensus.cc:3058] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 1 FOLLOWER]: Advancing to term 2
I20250815 19:59:07.752209 24358 raft_consensus.cc:513] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:59:07.752960 24358 leader_election.cc:304] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d87be07713104bb7828f7250198f837b; no voters:
I20250815 19:59:07.755230 24358 leader_election.cc:290] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 19:59:07.755613 24464 raft_consensus.cc:2802] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Leader election won for term 2
I20250815 19:59:07.759486 24358 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Time spent starting tablet: real 0.038s user 0.034s sys 0.002s
I20250815 19:59:07.759876 24315 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:35047
I20250815 19:59:07.760455 24468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:35047 every 8 connection(s)
I20250815 19:59:07.760798 24464 raft_consensus.cc:695] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 LEADER]: Becoming Leader. State: Replica: d87be07713104bb7828f7250198f837b, State: Running, Role: LEADER
I20250815 19:59:07.761554 24464 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } }
I20250815 19:59:07.762800 24315 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:59:07.767069 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 24315
I20250815 19:59:07.790140 24470 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:59:07.790614 24470 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:07.791567 24470 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:59:07.794936 23975 ts_manager.cc:194] Registered new tserver with Master: d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
I20250815 19:59:07.796278 23975 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b reported cstate change: term changed from 0 to 2, leader changed from <none> to d87be07713104bb7828f7250198f837b (127.20.53.67), VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) added. New cstate: current_term: 2 leader_uuid: "d87be07713104bb7828f7250198f837b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } health_report { overall_health: HEALTHY } } }
I20250815 19:59:07.807247 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:59:07.811020 23975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:47013
I20250815 19:59:07.815112 24470 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
I20250815 19:59:07.825747 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250815 19:59:07.829596 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250815 19:59:07.837229 24421 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } }
I20250815 19:59:07.840623 24464 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } }
I20250815 19:59:07.848325 23962 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9644c93dde714878a568a83a6bbd8a9d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 19:59:07.851805 23974 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b reported cstate change: config changed from index -1 to 9, NON_VOTER 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) added. New cstate: current_term: 2 leader_uuid: "d87be07713104bb7828f7250198f837b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 19:59:07.856628 24355 consensus_peers.cc:489] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b -> Peer 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217): Couldn't send request to peer 992c23efe942442db73e5fbcef76c9f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9644c93dde714878a568a83a6bbd8a9d. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:07.862509 24421 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:07.865973 24464 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } } }
W20250815 19:59:07.868332 24355 consensus_peers.cc:489] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b -> Peer 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217): Couldn't send request to peer 992c23efe942442db73e5fbcef76c9f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9644c93dde714878a568a83a6bbd8a9d. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:07.873878 23962 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9644c93dde714878a568a83a6bbd8a9d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250815 19:59:07.876686 24355 consensus_peers.cc:489] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b -> Peer d93df400fe844250aef143a864c15a9d (127.20.53.66:35271): Couldn't send request to peer d93df400fe844250aef143a864c15a9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9644c93dde714878a568a83a6bbd8a9d. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:07.876835 23974 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b reported cstate change: config changed from index 9 to 10, NON_VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) added. New cstate: current_term: 2 leader_uuid: "d87be07713104bb7828f7250198f837b" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 19:59:07.999495 24486 raft_consensus.cc:1062] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: attempting to promote NON_VOTER d93df400fe844250aef143a864c15a9d to VOTER
I20250815 19:59:08.000921 24486 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:08.005779 24261 raft_consensus.cc:1273] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Refusing update from remote peer 992c23efe942442db73e5fbcef76c9f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250815 19:59:08.007009 24485 consensus_queue.cc:1035] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250815 19:59:08.014516 24486 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:08.016201 24261 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:08.024864 23974 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index 8 to 9, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 19:59:08.035282 24117 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: NON_VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: true } }
I20250815 19:59:08.041510 24261 raft_consensus.cc:1273] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Refusing update from remote peer 992c23efe942442db73e5fbcef76c9f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250815 19:59:08.042785 24489 consensus_queue.cc:1035] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250815 19:59:08.048353 24486 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: NON_VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: true } } }
W20250815 19:59:08.050107 24051 consensus_peers.cc:489] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 -> Peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Couldn't send request to peer d87be07713104bb7828f7250198f837b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e3c262e14ae4134b7031e6a90066522. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:08.049801 24261 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: NON_VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: true } } }
I20250815 19:59:08.056532 23962 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e3c262e14ae4134b7031e6a90066522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 19:59:08.059569 23974 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index 9 to 10, NON_VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) added. New cstate: current_term: 2 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: NON_VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 19:59:08.402624 24493 ts_tablet_manager.cc:927] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Initiating tablet copy from peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
I20250815 19:59:08.404584 24493 tablet_copy_client.cc:323] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.53.67:35047
I20250815 19:59:08.406026 24441 tablet_copy_service.cc:140] P d87be07713104bb7828f7250198f837b: Received BeginTabletCopySession request for tablet 9644c93dde714878a568a83a6bbd8a9d from peer 992c23efe942442db73e5fbcef76c9f6 ({username='slave'} at 127.20.53.65:33727)
I20250815 19:59:08.406445 24441 tablet_copy_service.cc:161] P d87be07713104bb7828f7250198f837b: Beginning new tablet copy session on tablet 9644c93dde714878a568a83a6bbd8a9d from peer 992c23efe942442db73e5fbcef76c9f6 at {username='slave'} at 127.20.53.65:33727: session id = 992c23efe942442db73e5fbcef76c9f6-9644c93dde714878a568a83a6bbd8a9d
I20250815 19:59:08.411031 24441 tablet_copy_source_session.cc:215] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 19:59:08.414165 24493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9644c93dde714878a568a83a6bbd8a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:08.424173 24493 tablet_copy_client.cc:806] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: tablet copy: Starting download of 0 data blocks...
I20250815 19:59:08.424718 24493 tablet_copy_client.cc:670] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: tablet copy: Starting download of 1 WAL segments...
I20250815 19:59:08.428419 24493 tablet_copy_client.cc:538] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 19:59:08.433686 24493 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Bootstrap starting.
I20250815 19:59:08.434165 24495 ts_tablet_manager.cc:927] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Initiating tablet copy from peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
I20250815 19:59:08.435389 24495 tablet_copy_client.cc:323] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.53.67:35047
I20250815 19:59:08.442081 24498 ts_tablet_manager.cc:927] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Initiating tablet copy from peer 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217)
I20250815 19:59:08.444795 24498 tablet_copy_client.cc:323] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.53.65:40217
I20250815 19:59:08.445847 24441 tablet_copy_service.cc:140] P d87be07713104bb7828f7250198f837b: Received BeginTabletCopySession request for tablet 9644c93dde714878a568a83a6bbd8a9d from peer d93df400fe844250aef143a864c15a9d ({username='slave'} at 127.20.53.66:34309)
I20250815 19:59:08.446331 24441 tablet_copy_service.cc:161] P d87be07713104bb7828f7250198f837b: Beginning new tablet copy session on tablet 9644c93dde714878a568a83a6bbd8a9d from peer d93df400fe844250aef143a864c15a9d at {username='slave'} at 127.20.53.66:34309: session id = d93df400fe844250aef143a864c15a9d-9644c93dde714878a568a83a6bbd8a9d
I20250815 19:59:08.446484 24137 tablet_copy_service.cc:140] P 992c23efe942442db73e5fbcef76c9f6: Received BeginTabletCopySession request for tablet 6e3c262e14ae4134b7031e6a90066522 from peer d87be07713104bb7828f7250198f837b ({username='slave'} at 127.20.53.67:50627)
I20250815 19:59:08.446971 24137 tablet_copy_service.cc:161] P 992c23efe942442db73e5fbcef76c9f6: Beginning new tablet copy session on tablet 6e3c262e14ae4134b7031e6a90066522 from peer d87be07713104bb7828f7250198f837b at {username='slave'} at 127.20.53.67:50627: session id = d87be07713104bb7828f7250198f837b-6e3c262e14ae4134b7031e6a90066522
I20250815 19:59:08.450533 24441 tablet_copy_source_session.cc:215] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 19:59:08.452666 24495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9644c93dde714878a568a83a6bbd8a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:08.453909 24137 tablet_copy_source_session.cc:215] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 19:59:08.457105 24498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3c262e14ae4134b7031e6a90066522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:08.463303 24495 tablet_copy_client.cc:806] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: tablet copy: Starting download of 0 data blocks...
I20250815 19:59:08.463761 24495 tablet_copy_client.cc:670] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: tablet copy: Starting download of 1 WAL segments...
I20250815 19:59:08.466821 24495 tablet_copy_client.cc:538] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 19:59:08.471827 24495 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:59:08.475847 24498 tablet_copy_client.cc:806] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: tablet copy: Starting download of 0 data blocks...
I20250815 19:59:08.476486 24498 tablet_copy_client.cc:670] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: tablet copy: Starting download of 1 WAL segments...
I20250815 19:59:08.480768 24498 tablet_copy_client.cc:538] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 19:59:08.489498 24498 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Bootstrap starting.
I20250815 19:59:08.555608 24495 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:08.556195 24495 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Bootstrap complete.
I20250815 19:59:08.556625 24495 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.085s user 0.071s sys 0.012s
I20250815 19:59:08.558436 24495 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:08.558830 24493 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:08.558897 24495 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: LEARNER
I20250815 19:59:08.559317 24495 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:08.559574 24493 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Bootstrap complete.
I20250815 19:59:08.560081 24493 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Time spent bootstrapping tablet: real 0.127s user 0.106s sys 0.016s
I20250815 19:59:08.561378 24495 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250815 19:59:08.562263 24493 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:08.562999 24493 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Initialized, Role: LEARNER
I20250815 19:59:08.563086 24441 tablet_copy_service.cc:342] P d87be07713104bb7828f7250198f837b: Request end of tablet copy session d93df400fe844250aef143a864c15a9d-9644c93dde714878a568a83a6bbd8a9d received from {username='slave'} at 127.20.53.66:34309
I20250815 19:59:08.563493 24441 tablet_copy_service.cc:434] P d87be07713104bb7828f7250198f837b: ending tablet copy session d93df400fe844250aef143a864c15a9d-9644c93dde714878a568a83a6bbd8a9d on tablet 9644c93dde714878a568a83a6bbd8a9d with peer d93df400fe844250aef143a864c15a9d
I20250815 19:59:08.563575 24493 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: NON_VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: true } }
I20250815 19:59:08.565879 24493 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250815 19:59:08.566855 23961 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e3c262e14ae4134b7031e6a90066522 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250815 19:59:08.569015 24441 tablet_copy_service.cc:342] P d87be07713104bb7828f7250198f837b: Request end of tablet copy session 992c23efe942442db73e5fbcef76c9f6-9644c93dde714878a568a83a6bbd8a9d received from {username='slave'} at 127.20.53.65:33727
I20250815 19:59:08.569375 24441 tablet_copy_service.cc:434] P d87be07713104bb7828f7250198f837b: ending tablet copy session 992c23efe942442db73e5fbcef76c9f6-9644c93dde714878a568a83a6bbd8a9d on tablet 9644c93dde714878a568a83a6bbd8a9d with peer 992c23efe942442db73e5fbcef76c9f6
I20250815 19:59:08.589234 24498 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:08.589780 24498 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Bootstrap complete.
I20250815 19:59:08.590176 24498 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Time spent bootstrapping tablet: real 0.101s user 0.081s sys 0.017s
I20250815 19:59:08.591842 24498 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: NON_VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: true } }
I20250815 19:59:08.592278 24498 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d87be07713104bb7828f7250198f837b, State: Initialized, Role: LEARNER
I20250815 19:59:08.592644 24498 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: NON_VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: true } }
I20250815 19:59:08.594218 24498 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250815 19:59:08.596000 24137 tablet_copy_service.cc:342] P 992c23efe942442db73e5fbcef76c9f6: Request end of tablet copy session d87be07713104bb7828f7250198f837b-6e3c262e14ae4134b7031e6a90066522 received from {username='slave'} at 127.20.53.67:50627
I20250815 19:59:08.596414 24137 tablet_copy_service.cc:434] P 992c23efe942442db73e5fbcef76c9f6: ending tablet copy session d87be07713104bb7828f7250198f837b-6e3c262e14ae4134b7031e6a90066522 on tablet 6e3c262e14ae4134b7031e6a90066522 with peer d87be07713104bb7828f7250198f837b
I20250815 19:59:08.834049 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 992c23efe942442db73e5fbcef76c9f6 to finish bootstrapping
I20250815 19:59:08.848651 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d93df400fe844250aef143a864c15a9d to finish bootstrapping
I20250815 19:59:08.862874 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d87be07713104bb7828f7250198f837b to finish bootstrapping
I20250815 19:59:08.915683 24421 raft_consensus.cc:1215] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 19:59:08.955508 24117 raft_consensus.cc:1215] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 19:59:08.988327 24261 raft_consensus.cc:1215] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 19:59:09.127982 24097 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 19:59:09.130260 24401 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 19:59:09.135843 24241 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e50e1ff9f73d437eaf425225b75437c2 | 127.20.53.126:34801 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.53.84:33491 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
992c23efe942442db73e5fbcef76c9f6 | 127.20.53.65:40217 | HEALTHY | <none> | 1 | 0
d87be07713104bb7828f7250198f837b | 127.20.53.67:35047 | HEALTHY | <none> | 1 | 0
d93df400fe844250aef143a864c15a9d | 127.20.53.66:35271 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.53.65 | experimental | 127.20.53.65:40217
local_ip_for_outbound_sockets | 127.20.53.66 | experimental | 127.20.53.66:35271
local_ip_for_outbound_sockets | 127.20.53.67 | experimental | 127.20.53.67:35047
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.53.65:40217
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.53.66:35271
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.53.67:35047
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.53.84:33491 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 19:59:09.346096 20693 log_verifier.cc:126] Checking tablet 6e3c262e14ae4134b7031e6a90066522
I20250815 19:59:09.433677 20693 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 6e3c262e14ae4134b7031e6a90066522
I20250815 19:59:09.433974 20693 log_verifier.cc:126] Checking tablet 9644c93dde714878a568a83a6bbd8a9d
I20250815 19:59:09.494057 24517 raft_consensus.cc:1062] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: attempting to promote NON_VOTER d93df400fe844250aef143a864c15a9d to VOTER
I20250815 19:59:09.495645 24517 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:09.501087 24261 raft_consensus.cc:1273] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 LEARNER]: Refusing update from remote peer d87be07713104bb7828f7250198f837b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 19:59:09.502899 24117 raft_consensus.cc:1273] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEARNER]: Refusing update from remote peer d87be07713104bb7828f7250198f837b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 19:59:09.502616 24538 consensus_queue.cc:1035] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250815 19:59:09.503973 24517 consensus_queue.cc:1035] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Connected to new peer: Peer: permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 19:59:09.509631 24536 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:09.510836 24261 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:09.523737 23974 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b reported cstate change: config changed from index 10 to 11, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d87be07713104bb7828f7250198f837b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 19:59:09.525030 24117 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d93df400fe844250aef143a864c15a9d (127.20.53.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: NON_VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: true } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:09.526957 24536 raft_consensus.cc:1062] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: attempting to promote NON_VOTER 992c23efe942442db73e5fbcef76c9f6 to VOTER
I20250815 19:59:09.528465 24536 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:09.533794 20693 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 9644c93dde714878a568a83a6bbd8a9d
I20250815 19:59:09.534186 20693 log_verifier.cc:126] Checking tablet ab2b302e74d541578e701d63a5bbe304
I20250815 19:59:09.534422 24261 raft_consensus.cc:1273] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Refusing update from remote peer d87be07713104bb7828f7250198f837b: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 19:59:09.537109 24517 consensus_queue.cc:1035] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250815 19:59:09.543730 24116 raft_consensus.cc:1273] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEARNER]: Refusing update from remote peer d87be07713104bb7828f7250198f837b: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 19:59:09.547573 24517 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:09.549929 24541 raft_consensus.cc:1062] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: attempting to promote NON_VOTER d87be07713104bb7828f7250198f837b to VOTER
I20250815 19:59:09.549921 24261 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:09.551942 24541 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:09.550369 24543 consensus_queue.cc:1035] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [LEADER]: Connected to new peer: Peer: permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250815 19:59:09.559974 24116 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:09.581920 24421 raft_consensus.cc:1273] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 LEARNER]: Refusing update from remote peer 992c23efe942442db73e5fbcef76c9f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 19:59:09.582031 24261 raft_consensus.cc:1273] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Refusing update from remote peer 992c23efe942442db73e5fbcef76c9f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 19:59:09.583606 24515 consensus_queue.cc:1035] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250815 19:59:09.584429 24541 consensus_queue.cc:1035] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 19:59:09.584853 20693 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ab2b302e74d541578e701d63a5bbe304
I20250815 19:59:09.585158 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 23941
I20250815 19:59:09.592864 24541 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d87be07713104bb7828f7250198f837b (127.20.53.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } } }
I20250815 19:59:09.594378 24421 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d87be07713104bb7828f7250198f837b (127.20.53.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } } }
I20250815 19:59:09.596802 24261 raft_consensus.cc:2953] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d87be07713104bb7828f7250198f837b (127.20.53.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } } }
W20250815 19:59:09.620623 24162 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:34801 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.20.53.126:34801 (error 108)
W20250815 19:59:09.620739 24470 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:34801 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.20.53.126:34801 (error 108)
I20250815 19:59:09.623065 20693 minidump.cc:252] Setting minidump size limit to 20M
W20250815 19:59:09.624692 24311 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:34801 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.20.53.126:34801 (error 108)
I20250815 19:59:09.634377 20693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:09.636101 20693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:09.646132 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:09.646608 24557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:09.923660 24559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:09.924564 20693 server_base.cc:1047] running on GCE node
I20250815 19:59:09.926510 20693 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250815 19:59:09.926800 20693 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250815 19:59:09.927001 20693 hybrid_clock.cc:648] HybridClock initialized: now 1755287949926981 us; error 0 us; skew 500 ppm
I20250815 19:59:09.927811 20693 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:09.943969 20693 webserver.cc:480] Webserver started at http://0.0.0.0:33779/ using document root <none> and password file <none>
I20250815 19:59:09.945262 20693 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:09.945497 20693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:09.953735 20693 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250815 19:59:09.957769 24565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:09.958825 20693 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250815 19:59:09.959108 20693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e50e1ff9f73d437eaf425225b75437c2"
format_stamp: "Formatted at 2025-08-15 19:58:47 on dist-test-slave-d4bs"
I20250815 19:59:09.961237 20693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:09.975970 20693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:09.977344 20693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:09.977751 20693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:09.986325 20693 sys_catalog.cc:263] Verifying existing consensus state
W20250815 19:59:09.989622 20693 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.20.53.126:34801 exists but no master address supplied!
I20250815 19:59:09.991729 20693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap starting.
I20250815 19:59:10.033849 20693 log.cc:826] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:10.091753 20693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=13 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:10.092541 20693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap complete.
I20250815 19:59:10.106474 20693 raft_consensus.cc:357] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:10.107064 20693 raft_consensus.cc:738] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Initialized, Role: FOLLOWER
I20250815 19:59:10.107789 20693 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:10.108254 20693 raft_consensus.cc:397] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:10.108495 20693 raft_consensus.cc:491] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:10.108855 20693 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 3 FOLLOWER]: Advancing to term 4
I20250815 19:59:10.114054 20693 raft_consensus.cc:513] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:10.114715 20693 leader_election.cc:304] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e50e1ff9f73d437eaf425225b75437c2; no voters:
I20250815 19:59:10.115804 20693 leader_election.cc:290] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250815 19:59:10.116160 24572 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 19:59:10.117600 24572 raft_consensus.cc:695] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 4 LEADER]: Becoming Leader. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Running, Role: LEADER
I20250815 19:59:10.118527 24572 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:10.125200 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:59:10.125689 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:10.126816 24574 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50e1ff9f73d437eaf425225b75437c2. Latest consensus state: current_term: 4 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:59:10.127323 24574 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:10.152881 20693 tablet_replica.cc:331] stopping tablet replica
I20250815 19:59:10.153416 20693 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 4 LEADER]: Raft consensus shutting down.
I20250815 19:59:10.153796 20693 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250815 19:59:10.156148 20693 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 19:59:10.156638 20693 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 19:59:10.180640 20693 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250815 19:59:15.312623 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 24012
I20250815 19:59:15.338460 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 24167
I20250815 19:59:15.364691 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 24315
I20250815 19:59:15.394476 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:34801
--webserver_interface=127.20.53.126
--webserver_port=42575
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:34801 with env {}
W20250815 19:59:15.703459 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:15.704031 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:15.704530 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:15.735188 24646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:59:15.735534 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:15.735795 24646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:59:15.736042 24646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:59:15.770880 24646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:34801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=42575
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:15.772321 24646 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:15.773932 24646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:15.783864 24652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:17.190135 24651 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 24646
W20250815 19:59:17.561543 24651 kernel_stack_watchdog.cc:198] Thread 24646 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:59:15.785817 24653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:17.562341 24646 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.777s user 0.000s sys 0.002s
W20250815 19:59:17.562690 24646 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.778s user 0.000s sys 0.002s
W20250815 19:59:17.563026 24654 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1777 milliseconds
I20250815 19:59:17.564342 24646 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 19:59:17.564363 24655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:17.567345 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:17.569790 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:17.571144 24646 hybrid_clock.cc:648] HybridClock initialized: now 1755287957571095 us; error 47 us; skew 500 ppm
I20250815 19:59:17.571931 24646 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:17.577867 24646 webserver.cc:480] Webserver started at http://127.20.53.126:42575/ using document root <none> and password file <none>
I20250815 19:59:17.578753 24646 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:17.578948 24646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:17.586709 24646 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250815 19:59:17.591307 24662 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:17.592619 24646 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 19:59:17.592985 24646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e50e1ff9f73d437eaf425225b75437c2"
format_stamp: "Formatted at 2025-08-15 19:58:47 on dist-test-slave-d4bs"
I20250815 19:59:17.594909 24646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:17.665086 24646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:17.666514 24646 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:17.666918 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:17.736601 24646 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:34801
I20250815 19:59:17.736797 24713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:34801 every 8 connection(s)
I20250815 19:59:17.739554 24646 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:59:17.749641 24714 sys_catalog.cc:263] Verifying existing consensus state
I20250815 19:59:17.750134 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 24646
I20250815 19:59:17.751581 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:40217
--local_ip_for_outbound_sockets=127.20.53.65
--tserver_master_addrs=127.20.53.126:34801
--webserver_port=43373
--webserver_interface=127.20.53.65
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:59:17.754801 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap starting.
I20250815 19:59:17.767551 24714 log.cc:826] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:17.866411 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=15 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:17.867213 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Bootstrap complete.
I20250815 19:59:17.885548 24714 raft_consensus.cc:357] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:17.887657 24714 raft_consensus.cc:738] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Initialized, Role: FOLLOWER
I20250815 19:59:17.888530 24714 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:17.889016 24714 raft_consensus.cc:397] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:17.889272 24714 raft_consensus.cc:491] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:17.889559 24714 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 5 FOLLOWER]: Advancing to term 6
I20250815 19:59:17.894511 24714 raft_consensus.cc:513] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:17.895098 24714 leader_election.cc:304] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e50e1ff9f73d437eaf425225b75437c2; no voters:
I20250815 19:59:17.897233 24714 leader_election.cc:290] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250815 19:59:17.897554 24718 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 6 FOLLOWER]: Leader election won for term 6
I20250815 19:59:17.900691 24718 raft_consensus.cc:695] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [term 6 LEADER]: Becoming Leader. State: Replica: e50e1ff9f73d437eaf425225b75437c2, State: Running, Role: LEADER
I20250815 19:59:17.901566 24718 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } }
I20250815 19:59:17.902177 24714 sys_catalog.cc:564] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:59:17.911593 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:59:17.912410 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:17.914825 24720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50e1ff9f73d437eaf425225b75437c2. Latest consensus state: current_term: 6 leader_uuid: "e50e1ff9f73d437eaf425225b75437c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50e1ff9f73d437eaf425225b75437c2" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 34801 } } }
I20250815 19:59:17.915562 24720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2 [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:17.924352 24726 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:59:17.937451 24726 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=192f2dcb486249c59cb348216a45965d]
I20250815 19:59:17.939126 24726 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=3bb3a8ea90fb455dbd6f289280f383dc]
I20250815 19:59:17.940666 24726 catalog_manager.cc:671] Loaded metadata for table TestTable [id=664b23e1c5c84caf8c01d0dbf3ecdb7c]
I20250815 19:59:17.948998 24726 tablet_loader.cc:96] loaded metadata for tablet 6e3c262e14ae4134b7031e6a90066522 (table TestTable1 [id=3bb3a8ea90fb455dbd6f289280f383dc])
I20250815 19:59:17.950275 24726 tablet_loader.cc:96] loaded metadata for tablet 9644c93dde714878a568a83a6bbd8a9d (table TestTable [id=664b23e1c5c84caf8c01d0dbf3ecdb7c])
I20250815 19:59:17.951633 24726 tablet_loader.cc:96] loaded metadata for tablet ab2b302e74d541578e701d63a5bbe304 (table TestTable2 [id=192f2dcb486249c59cb348216a45965d])
I20250815 19:59:17.953210 24726 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:59:17.959040 24726 catalog_manager.cc:1261] Loaded cluster ID: 2ebeb2378ff247dca7924065e794a7d6
I20250815 19:59:17.959342 24726 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:59:17.967880 24726 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:59:17.969702 24735 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250815 19:59:17.973889 24726 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e50e1ff9f73d437eaf425225b75437c2: Loaded TSK: 0
I20250815 19:59:17.975373 24726 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 19:59:18.113029 24716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:18.113556 24716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:18.114053 24716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:18.145290 24716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:18.146112 24716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:59:18.181694 24716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:40217
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=43373
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:18.183022 24716 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:18.184592 24716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:18.197490 24741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:18.201954 24744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:18.201280 24742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:19.669175 24743 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1467 milliseconds
I20250815 19:59:19.669314 24716 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:19.670756 24716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:19.673569 24716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:19.675062 24716 hybrid_clock.cc:648] HybridClock initialized: now 1755287959675008 us; error 58 us; skew 500 ppm
I20250815 19:59:19.676093 24716 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:19.683481 24716 webserver.cc:480] Webserver started at http://127.20.53.65:43373/ using document root <none> and password file <none>
I20250815 19:59:19.684674 24716 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:19.684971 24716 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:19.695266 24716 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20250815 19:59:19.701154 24751 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:19.702276 24716 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250815 19:59:19.702697 24716 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "992c23efe942442db73e5fbcef76c9f6"
format_stamp: "Formatted at 2025-08-15 19:58:49 on dist-test-slave-d4bs"
I20250815 19:59:19.705651 24716 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:19.764918 24716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:19.766320 24716 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:19.766736 24716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:19.769738 24716 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:19.776046 24758 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 19:59:19.788247 24716 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 19:59:19.788507 24716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20250815 19:59:19.788868 24716 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 19:59:19.794304 24758 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Bootstrap starting.
I20250815 19:59:19.799276 24716 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 19:59:19.799561 24716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.008s sys 0.000s
I20250815 19:59:19.877156 24758 log.cc:826] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:20.031213 24716 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:40217
I20250815 19:59:20.031428 24866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:40217 every 8 connection(s)
I20250815 19:59:20.035094 24716 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:59:20.038570 24758 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:20.039732 24758 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Bootstrap complete.
I20250815 19:59:20.042042 24758 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Time spent bootstrapping tablet: real 0.248s user 0.196s sys 0.045s
I20250815 19:59:20.042243 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 24716
I20250815 19:59:20.044878 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:35271
--local_ip_for_outbound_sockets=127.20.53.66
--tserver_master_addrs=127.20.53.126:34801
--webserver_port=35999
--webserver_interface=127.20.53.66
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:59:20.082389 24758 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:20.086391 24758 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Initialized, Role: FOLLOWER
I20250815 19:59:20.087482 24758 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:20.091528 24758 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6: Time spent starting tablet: real 0.049s user 0.024s sys 0.003s
I20250815 19:59:20.092270 24758 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap starting.
I20250815 19:59:20.106591 24867 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:59:20.107076 24867 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:20.108342 24867 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:59:20.114977 24679 ts_manager.cc:194] Registered new tserver with Master: 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217)
I20250815 19:59:20.120244 24679 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) added, VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) added, VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:20.122279 24679 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index 10 to 11, leader changed from 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65) to <none>, d87be07713104bb7828f7250198f837b (127.20.53.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } } }
I20250815 19:59:20.203095 24679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:46447
I20250815 19:59:20.208007 24867 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
I20250815 19:59:20.240736 24758 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:20.241614 24758 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Bootstrap complete.
I20250815 19:59:20.243227 24758 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Time spent bootstrapping tablet: real 0.151s user 0.130s sys 0.016s
I20250815 19:59:20.245365 24758 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:20.246011 24758 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Initialized, Role: FOLLOWER
I20250815 19:59:20.246619 24758 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:20.248524 24758 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20250815 19:59:20.414564 24871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:20.415068 24871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:20.415566 24871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:20.447379 24871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:20.448278 24871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:59:20.483536 24871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:35271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=35999
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:20.484939 24871 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:20.486503 24871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:20.498842 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:21.584810 24889 raft_consensus.cc:491] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:59:21.585907 24889 raft_consensus.cc:513] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:21.598489 24889 leader_election.cc:290] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d87be07713104bb7828f7250198f837b (127.20.53.67:35047), d93df400fe844250aef143a864c15a9d (127.20.53.66:35271)
W20250815 19:59:21.606359 24755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
W20250815 19:59:21.621688 24755 leader_election.cc:336] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
W20250815 19:59:21.626003 24755 leader_election.cc:336] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d93df400fe844250aef143a864c15a9d (127.20.53.66:35271): Network error: Client connection negotiation failed: client connection to 127.20.53.66:35271: connect: Connection refused (error 111)
I20250815 19:59:21.626674 24755 leader_election.cc:304] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6; no voters: d87be07713104bb7828f7250198f837b, d93df400fe844250aef143a864c15a9d
I20250815 19:59:21.627811 24889 raft_consensus.cc:2747] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 19:59:22.055115 24889 raft_consensus.cc:491] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:59:22.055724 24889 raft_consensus.cc:513] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:22.058539 24889 leader_election.cc:290] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d93df400fe844250aef143a864c15a9d (127.20.53.66:35271), d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
W20250815 19:59:22.070405 24755 leader_election.cc:336] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d93df400fe844250aef143a864c15a9d (127.20.53.66:35271): Network error: Client connection negotiation failed: client connection to 127.20.53.66:35271: connect: Connection refused (error 111)
W20250815 19:59:22.071367 24755 leader_election.cc:336] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
I20250815 19:59:22.071928 24755 leader_election.cc:304] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6; no voters: d87be07713104bb7828f7250198f837b, d93df400fe844250aef143a864c15a9d
I20250815 19:59:22.072928 24889 raft_consensus.cc:2747] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250815 19:59:21.903422 24882 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24871
W20250815 19:59:22.236279 24882 kernel_stack_watchdog.cc:198] Thread 24871 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:59:20.500857 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:22.237195 24871 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.737s user 0.617s sys 1.099s
W20250815 19:59:22.237576 24871 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.738s user 0.617s sys 1.099s
W20250815 19:59:22.237684 24885 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1736 milliseconds
W20250815 19:59:22.238448 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:22.238412 24871 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:22.241623 24871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:22.243606 24871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:22.244943 24871 hybrid_clock.cc:648] HybridClock initialized: now 1755287962244909 us; error 38 us; skew 500 ppm
I20250815 19:59:22.245734 24871 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:22.251499 24871 webserver.cc:480] Webserver started at http://127.20.53.66:35999/ using document root <none> and password file <none>
I20250815 19:59:22.252394 24871 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:22.252629 24871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:22.260078 24871 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250815 19:59:22.264856 24897 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:22.266016 24871 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250815 19:59:22.266426 24871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d93df400fe844250aef143a864c15a9d"
format_stamp: "Formatted at 2025-08-15 19:58:51 on dist-test-slave-d4bs"
I20250815 19:59:22.268545 24871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:22.312107 24871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:22.313597 24871 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:22.314033 24871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:22.316422 24871 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:22.321848 24904 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250815 19:59:22.336546 24871 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250815 19:59:22.336786 24871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.002s
I20250815 19:59:22.336998 24871 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250815 19:59:22.341850 24904 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:59:22.347584 24871 ts_tablet_manager.cc:610] Registered 3 tablets
I20250815 19:59:22.347774 24871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.008s sys 0.000s
I20250815 19:59:22.393103 24904 log.cc:826] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:22.508394 24871 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:35271
I20250815 19:59:22.508486 25011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:35271 every 8 connection(s)
I20250815 19:59:22.511801 24871 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:59:22.512820 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 24871
I20250815 19:59:22.514792 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:35047
--local_ip_for_outbound_sockets=127.20.53.67
--tserver_master_addrs=127.20.53.126:34801
--webserver_port=33979
--webserver_interface=127.20.53.67
--builtin_ntp_servers=127.20.53.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 19:59:22.539067 24904 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:22.539816 24904 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Bootstrap complete.
I20250815 19:59:22.541204 24904 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.200s user 0.164s sys 0.032s
I20250815 19:59:22.552896 25012 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:59:22.553400 25012 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:22.554627 25012 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:59:22.554391 24904 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:22.556950 24904 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: FOLLOWER
I20250815 19:59:22.557606 24904 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:22.561517 24904 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20250815 19:59:22.562155 24904 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:59:22.566607 24679 ts_manager.cc:194] Registered new tserver with Master: d93df400fe844250aef143a864c15a9d (127.20.53.66:35271)
I20250815 19:59:22.572028 24679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:49573
I20250815 19:59:22.575567 25012 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
I20250815 19:59:22.654531 24904 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:22.655262 24904 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Bootstrap complete.
I20250815 19:59:22.656405 24904 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.094s user 0.075s sys 0.015s
I20250815 19:59:22.657939 24904 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:22.658403 24904 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: FOLLOWER
I20250815 19:59:22.658869 24904 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:22.660167 24904 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250815 19:59:22.660761 24904 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap starting.
I20250815 19:59:22.744763 24904 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:22.745674 24904 tablet_bootstrap.cc:492] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Bootstrap complete.
I20250815 19:59:22.747166 24904 ts_tablet_manager.cc:1397] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Time spent bootstrapping tablet: real 0.087s user 0.080s sys 0.004s
I20250815 19:59:22.749080 24904 raft_consensus.cc:357] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:22.749532 24904 raft_consensus.cc:738] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93df400fe844250aef143a864c15a9d, State: Initialized, Role: FOLLOWER
I20250815 19:59:22.750062 24904 consensus_queue.cc:260] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:22.750620 24904 raft_consensus.cc:397] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:22.750927 24904 raft_consensus.cc:491] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:22.751336 24904 raft_consensus.cc:3058] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Advancing to term 3
I20250815 19:59:22.759107 24904 raft_consensus.cc:513] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:22.759985 24904 leader_election.cc:304] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d93df400fe844250aef143a864c15a9d; no voters:
I20250815 19:59:22.760635 24904 leader_election.cc:290] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 19:59:22.760860 25017 raft_consensus.cc:2802] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Leader election won for term 3
I20250815 19:59:22.766088 24904 ts_tablet_manager.cc:1428] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20250815 19:59:22.773806 25017 raft_consensus.cc:695] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [term 3 LEADER]: Becoming Leader. State: Replica: d93df400fe844250aef143a864c15a9d, State: Running, Role: LEADER
I20250815 19:59:22.774704 25017 consensus_queue.cc:237] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } }
I20250815 19:59:22.782591 24679 catalog_manager.cc:5582] T ab2b302e74d541578e701d63a5bbe304 P d93df400fe844250aef143a864c15a9d reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "d93df400fe844250aef143a864c15a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } health_report { overall_health: HEALTHY } } }
W20250815 19:59:22.863977 25016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:22.864462 25016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:22.864997 25016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:22.895619 25016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:22.896428 25016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:59:22.931025 25016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:35047
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=33979
--tserver_master_addrs=127.20.53.126:34801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:22.932315 25016 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:22.933931 25016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:22.947800 25033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:23.807040 25039 raft_consensus.cc:491] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:59:23.807791 25039 raft_consensus.cc:513] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:23.818583 25039 leader_election.cc:290] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d87be07713104bb7828f7250198f837b (127.20.53.67:35047), d93df400fe844250aef143a864c15a9d (127.20.53.66:35271)
W20250815 19:59:23.846876 24755 leader_election.cc:336] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
I20250815 19:59:23.866442 24967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" candidate_uuid: "992c23efe942442db73e5fbcef76c9f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d93df400fe844250aef143a864c15a9d" is_pre_election: true
I20250815 19:59:23.867527 24967 raft_consensus.cc:2466] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 992c23efe942442db73e5fbcef76c9f6 in term 2.
I20250815 19:59:23.869626 24755 leader_election.cc:304] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6, d93df400fe844250aef143a864c15a9d; no voters: d87be07713104bb7828f7250198f837b
I20250815 19:59:23.870774 25039 raft_consensus.cc:2802] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 19:59:23.871305 25039 raft_consensus.cc:491] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:59:23.871783 25039 raft_consensus.cc:3058] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Advancing to term 3
I20250815 19:59:23.886014 25039 raft_consensus.cc:513] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:23.889207 25039 leader_election.cc:290] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 election: Requested vote from peers d87be07713104bb7828f7250198f837b (127.20.53.67:35047), d93df400fe844250aef143a864c15a9d (127.20.53.66:35271)
I20250815 19:59:23.891420 24967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" candidate_uuid: "992c23efe942442db73e5fbcef76c9f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d93df400fe844250aef143a864c15a9d"
I20250815 19:59:23.892410 24967 raft_consensus.cc:3058] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Advancing to term 3
W20250815 19:59:23.895413 24755 leader_election.cc:336] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
I20250815 19:59:23.902534 24967 raft_consensus.cc:2466] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992c23efe942442db73e5fbcef76c9f6 in term 3.
I20250815 19:59:23.904032 24755 leader_election.cc:304] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6, d93df400fe844250aef143a864c15a9d; no voters: d87be07713104bb7828f7250198f837b
I20250815 19:59:23.905151 25039 raft_consensus.cc:2802] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 19:59:23.907851 25039 raft_consensus.cc:695] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 3 LEADER]: Becoming Leader. State: Replica: 992c23efe942442db73e5fbcef76c9f6, State: Running, Role: LEADER
I20250815 19:59:23.909569 25039 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:23.929780 24679 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: term changed from 2 to 3, leader changed from <none> to 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65). New cstate: current_term: 3 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250815 19:59:23.966735 25050 raft_consensus.cc:491] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:59:23.967518 25050 raft_consensus.cc:513] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:23.971580 25050 leader_election.cc:290] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217), d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
W20250815 19:59:23.996011 24901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
I20250815 19:59:24.009168 24821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e3c262e14ae4134b7031e6a90066522" candidate_uuid: "d93df400fe844250aef143a864c15a9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "992c23efe942442db73e5fbcef76c9f6" is_pre_election: true
W20250815 19:59:24.010113 24901 leader_election.cc:336] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
I20250815 19:59:24.010279 24821 raft_consensus.cc:2466] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93df400fe844250aef143a864c15a9d in term 2.
I20250815 19:59:24.012161 24901 leader_election.cc:304] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6, d93df400fe844250aef143a864c15a9d; no voters: d87be07713104bb7828f7250198f837b
I20250815 19:59:24.013163 25050 raft_consensus.cc:2802] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 19:59:24.013512 25050 raft_consensus.cc:491] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:59:24.013820 25050 raft_consensus.cc:3058] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 2 FOLLOWER]: Advancing to term 3
I20250815 19:59:24.020573 25050 raft_consensus.cc:513] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:24.022394 25050 leader_election.cc:290] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 election: Requested vote from peers 992c23efe942442db73e5fbcef76c9f6 (127.20.53.65:40217), d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
I20250815 19:59:24.023530 24821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e3c262e14ae4134b7031e6a90066522" candidate_uuid: "d93df400fe844250aef143a864c15a9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "992c23efe942442db73e5fbcef76c9f6"
I20250815 19:59:24.024143 24821 raft_consensus.cc:3058] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 2 FOLLOWER]: Advancing to term 3
W20250815 19:59:24.029325 24901 leader_election.cc:336] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111)
I20250815 19:59:24.036870 24821 raft_consensus.cc:2466] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93df400fe844250aef143a864c15a9d in term 3.
I20250815 19:59:24.038465 24901 leader_election.cc:304] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 992c23efe942442db73e5fbcef76c9f6, d93df400fe844250aef143a864c15a9d; no voters: d87be07713104bb7828f7250198f837b
I20250815 19:59:24.040948 25050 raft_consensus.cc:2802] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Leader election won for term 3
I20250815 19:59:24.049681 25050 raft_consensus.cc:695] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [term 3 LEADER]: Becoming Leader. State: Replica: d93df400fe844250aef143a864c15a9d, State: Running, Role: LEADER
I20250815 19:59:24.050516 25050 consensus_queue.cc:237] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:24.065929 24679 catalog_manager.cc:5582] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d reported cstate change: term changed from 2 to 3, leader changed from <none> to d93df400fe844250aef143a864c15a9d (127.20.53.66). New cstate: current_term: 3 leader_uuid: "d93df400fe844250aef143a864c15a9d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250815 19:59:22.949783 25034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:24.225069 25036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:24.227088 25035 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1273 milliseconds
I20250815 19:59:24.227178 25016 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:24.228364 25016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:24.230589 25016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:24.231927 25016 hybrid_clock.cc:648] HybridClock initialized: now 1755287964231879 us; error 47 us; skew 500 ppm
I20250815 19:59:24.232743 25016 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:24.239143 25016 webserver.cc:480] Webserver started at http://127.20.53.67:33979/ using document root <none> and password file <none>
I20250815 19:59:24.240008 25016 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:24.240219 25016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:24.247722 25016 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20250815 19:59:24.252559 25061 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:24.253611 25016 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 19:59:24.253930 25016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d87be07713104bb7828f7250198f837b"
format_stamp: "Formatted at 2025-08-15 19:58:53 on dist-test-slave-d4bs"
I20250815 19:59:24.255704 25016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:24.301194 24967 raft_consensus.cc:1273] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Refusing update from remote peer 992c23efe942442db73e5fbcef76c9f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250815 19:59:24.302693 25039 consensus_queue.cc:1035] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250815 19:59:24.307628 25016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:24.309782 25016 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:24.310345 25016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:24.319666 25016 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250815 19:59:24.329773 24755 consensus_peers.cc:489] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 -> Peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Couldn't send request to peer d87be07713104bb7828f7250198f837b. Status: Network error: Client connection negotiation failed: client connection to 127.20.53.67:35047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:24.330013 25072 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 19:59:24.344902 24821 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:24.345638 25016 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 19:59:24.345942 25016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.003s sys 0.000s
I20250815 19:59:24.346302 25016 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 19:59:24.349078 24967 raft_consensus.cc:1273] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Refusing update from remote peer 992c23efe942442db73e5fbcef76c9f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250815 19:59:24.350273 25070 consensus_queue.cc:1035] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250815 19:59:24.353394 25072 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap starting.
I20250815 19:59:24.356339 25044 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:24.358100 25016 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 19:59:24.358345 25016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.007s sys 0.003s
I20250815 19:59:24.357852 24967 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } }
I20250815 19:59:24.366531 24666 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9644c93dde714878a568a83a6bbd8a9d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 19:59:24.372440 24679 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index 12 to 14, VOTER d87be07713104bb7828f7250198f837b (127.20.53.67) evicted. New cstate: current_term: 3 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250815 19:59:24.382257 24679 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 9644c93dde714878a568a83a6bbd8a9d on TS d87be07713104bb7828f7250198f837b: Not found: failed to reset TS proxy: Could not find TS for UUID d87be07713104bb7828f7250198f837b
I20250815 19:59:24.387269 24821 consensus_queue.cc:237] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } }
I20250815 19:59:24.390075 25044 raft_consensus.cc:2953] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } }
I20250815 19:59:24.399046 24666 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9644c93dde714878a568a83a6bbd8a9d with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 19:59:24.403692 24678 catalog_manager.cc:5582] T 9644c93dde714878a568a83a6bbd8a9d P 992c23efe942442db73e5fbcef76c9f6 reported cstate change: config changed from index 14 to 15, VOTER d93df400fe844250aef143a864c15a9d (127.20.53.66) evicted. New cstate: current_term: 3 leader_uuid: "992c23efe942442db73e5fbcef76c9f6" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 19:59:24.419891 25072 log.cc:826] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:24.431876 24947 tablet_service.cc:1515] Processing DeleteTablet for tablet 9644c93dde714878a568a83a6bbd8a9d with delete_type TABLET_DATA_TOMBSTONED (TS d93df400fe844250aef143a864c15a9d not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:56760
W20250815 19:59:24.435678 24664 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 9644c93dde714878a568a83a6bbd8a9d on TS d87be07713104bb7828f7250198f837b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d87be07713104bb7828f7250198f837b
I20250815 19:59:24.438550 25099 tablet_replica.cc:331] stopping tablet replica
I20250815 19:59:24.439522 25099 raft_consensus.cc:2241] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Raft consensus shutting down.
I20250815 19:59:24.440305 25099 raft_consensus.cc:2270] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250815 19:59:24.444751 25099 ts_tablet_manager.cc:1905] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 19:59:24.464156 25099 ts_tablet_manager.cc:1918] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250815 19:59:24.464710 25099 log.cc:1199] T 9644c93dde714878a568a83a6bbd8a9d P d93df400fe844250aef143a864c15a9d: Deleting WAL directory at /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/wals/9644c93dde714878a568a83a6bbd8a9d
I20250815 19:59:24.466554 24666 catalog_manager.cc:4928] TS d93df400fe844250aef143a864c15a9d (127.20.53.66:35271): tablet 9644c93dde714878a568a83a6bbd8a9d (table TestTable [id=664b23e1c5c84caf8c01d0dbf3ecdb7c]) successfully deleted
I20250815 19:59:24.567400 25072 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:24.568369 25016 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:35047
I20250815 19:59:24.568519 25182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:35047 every 8 connection(s)
I20250815 19:59:24.568913 25072 tablet_bootstrap.cc:492] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Bootstrap complete.
I20250815 19:59:24.570220 25072 ts_tablet_manager.cc:1397] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Time spent bootstrapping tablet: real 0.217s user 0.176s sys 0.031s
I20250815 19:59:24.571813 25016 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:59:24.578847 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25016
I20250815 19:59:24.605566 25072 raft_consensus.cc:357] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:24.608975 25072 raft_consensus.cc:738] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87be07713104bb7828f7250198f837b, State: Initialized, Role: FOLLOWER
I20250815 19:59:24.609937 25072 consensus_queue.cc:260] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } } peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } attrs { promote: false } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } }
I20250815 19:59:24.614823 25072 ts_tablet_manager.cc:1428] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Time spent starting tablet: real 0.044s user 0.028s sys 0.005s
I20250815 19:59:24.615517 25072 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Bootstrap starting.
I20250815 19:59:24.622484 25183 heartbeater.cc:344] Connected to a master server at 127.20.53.126:34801
I20250815 19:59:24.622960 25183 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:24.623976 25183 heartbeater.cc:507] Master 127.20.53.126:34801 requested a full tablet report, sending...
I20250815 19:59:24.627629 24679 ts_manager.cc:194] Registered new tserver with Master: d87be07713104bb7828f7250198f837b (127.20.53.67:35047)
I20250815 19:59:24.632256 24679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:50013
I20250815 19:59:24.637185 25183 heartbeater.cc:499] Master 127.20.53.126:34801 was elected leader, sending a full tablet report...
I20250815 19:59:24.640453 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 19:59:24.644647 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250815 19:59:24.648465 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250815 19:59:24.655941 24821 raft_consensus.cc:1273] T 6e3c262e14ae4134b7031e6a90066522 P 992c23efe942442db73e5fbcef76c9f6 [term 3 FOLLOWER]: Refusing update from remote peer d93df400fe844250aef143a864c15a9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250815 19:59:24.657289 25050 consensus_queue.cc:1035] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250815 19:59:24.692219 24901 consensus_peers.cc:489] T 6e3c262e14ae4134b7031e6a90066522 P d93df400fe844250aef143a864c15a9d -> Peer d87be07713104bb7828f7250198f837b (127.20.53.67:35047): Couldn't send request to peer d87be07713104bb7828f7250198f837b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:24.723114 25072 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 19:59:24.723814 25072 tablet_bootstrap.cc:492] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Bootstrap complete.
I20250815 19:59:24.725001 25072 ts_tablet_manager.cc:1397] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Time spent bootstrapping tablet: real 0.110s user 0.088s sys 0.019s
I20250815 19:59:24.726644 25072 raft_consensus.cc:357] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:24.727075 25072 raft_consensus.cc:738] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87be07713104bb7828f7250198f837b, State: Initialized, Role: FOLLOWER
I20250815 19:59:24.727646 25072 consensus_queue.cc:260] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "992c23efe942442db73e5fbcef76c9f6" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 40217 } } peers { permanent_uuid: "d93df400fe844250aef143a864c15a9d" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 35271 } attrs { promote: false } } peers { permanent_uuid: "d87be07713104bb7828f7250198f837b" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 35047 } attrs { promote: false } }
I20250815 19:59:24.728955 25072 ts_tablet_manager.cc:1428] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250815 19:59:24.781535 25118 tablet_service.cc:1515] Processing DeleteTablet for tablet 9644c93dde714878a568a83a6bbd8a9d with delete_type TABLET_DATA_TOMBSTONED (TS d87be07713104bb7828f7250198f837b not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:51458
I20250815 19:59:24.783970 25196 tablet_replica.cc:331] stopping tablet replica
I20250815 19:59:24.784581 25196 raft_consensus.cc:2241] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 19:59:24.785009 25196 raft_consensus.cc:2270] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 19:59:24.787434 25196 ts_tablet_manager.cc:1905] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 19:59:24.799700 25196 ts_tablet_manager.cc:1918] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250815 19:59:24.799976 25196 log.cc:1199] T 9644c93dde714878a568a83a6bbd8a9d P d87be07713104bb7828f7250198f837b: Deleting WAL directory at /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/wals/9644c93dde714878a568a83a6bbd8a9d
I20250815 19:59:24.801446 24666 catalog_manager.cc:4928] TS d87be07713104bb7828f7250198f837b (127.20.53.67:35047): tablet 9644c93dde714878a568a83a6bbd8a9d (table TestTable [id=664b23e1c5c84caf8c01d0dbf3ecdb7c]) successfully deleted
I20250815 19:59:25.252398 25138 raft_consensus.cc:3058] T 6e3c262e14ae4134b7031e6a90066522 P d87be07713104bb7828f7250198f837b [term 2 FOLLOWER]: Advancing to term 3
W20250815 19:59:25.652491 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:26.656227 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:27.659930 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:28.663532 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:29.666744 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:30.670922 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:30.904924 24863 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7140 after lost signal to thread 24736
W20250815 19:59:30.905803 24863 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5760 after lost signal to thread 24866
W20250815 19:59:31.675062 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:32.678278 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:33.681639 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:34.685134 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:35.688611 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:36.691989 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:37.695358 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:38.698583 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:39.702286 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:40.706367 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:41.710268 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:42.713557 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 19:59:43.716837 20693 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9644c93dde714878a568a83a6bbd8a9d: tablet_id: "9644c93dde714878a568a83a6bbd8a9d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250815 19:59:44.719895 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 24716
I20250815 19:59:44.750070 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 24871
I20250815 19:59:44.782464 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25016
I20250815 19:59:44.808652 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 24646
2025-08-15T19:59:44Z chronyd exiting
I20250815 19:59:44.858291 20693 test_util.cc:183] -----------------------------------------------
I20250815 19:59:44.858501 20693 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755287859650099-20693-0
[ FAILED ] AdminCliTest.TestRebuildTables (59074 ms)
[----------] 5 tests from AdminCliTest (125137 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250815 19:59:44.862385 20693 test_util.cc:276] Using random seed: -724503400
I20250815 19:59:44.866714 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 19:59:44.866870 20693 ts_itest-base.cc:116] --------------
I20250815 19:59:44.867043 20693 ts_itest-base.cc:117] 5 tablet servers
I20250815 19:59:44.867187 20693 ts_itest-base.cc:118] 3 replicas per TS
I20250815 19:59:44.867316 20693 ts_itest-base.cc:119] --------------
2025-08-15T19:59:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T19:59:44Z Disabled control of system clock
I20250815 19:59:44.910912 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:39169
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:39169
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 19:59:45.215690 25224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:45.216312 25224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:45.216810 25224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:45.247249 25224 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 19:59:45.247634 25224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 19:59:45.247905 25224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:45.248147 25224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 19:59:45.248385 25224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 19:59:45.283182 25224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:39169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:45.284507 25224 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:45.286125 25224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:45.297765 25230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:45.298374 25231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:46.487980 25233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:46.490705 25232 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1188 milliseconds
I20250815 19:59:46.490864 25224 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:46.492131 25224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:46.494736 25224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:46.496078 25224 hybrid_clock.cc:648] HybridClock initialized: now 1755287986496057 us; error 38 us; skew 500 ppm
I20250815 19:59:46.496906 25224 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:46.503468 25224 webserver.cc:480] Webserver started at http://127.20.53.126:33937/ using document root <none> and password file <none>
I20250815 19:59:46.504319 25224 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:46.504520 25224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:46.504971 25224 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:59:46.509315 25224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "46abfa7a5e394d71a5c55cfa384c532c"
format_stamp: "Formatted at 2025-08-15 19:59:46 on dist-test-slave-d4bs"
I20250815 19:59:46.510396 25224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "46abfa7a5e394d71a5c55cfa384c532c"
format_stamp: "Formatted at 2025-08-15 19:59:46 on dist-test-slave-d4bs"
I20250815 19:59:46.517320 25224 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250815 19:59:46.522828 25240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:46.523860 25224 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250815 19:59:46.524171 25224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "46abfa7a5e394d71a5c55cfa384c532c"
format_stamp: "Formatted at 2025-08-15 19:59:46 on dist-test-slave-d4bs"
I20250815 19:59:46.524489 25224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:46.587548 25224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:46.589041 25224 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:46.589500 25224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:46.659927 25224 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:39169
I20250815 19:59:46.660009 25291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:39169 every 8 connection(s)
I20250815 19:59:46.662631 25224 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 19:59:46.667639 25292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:46.671852 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25224
I20250815 19:59:46.672407 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 19:59:46.691857 25292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c: Bootstrap starting.
I20250815 19:59:46.698499 25292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c: Neither blocks nor log segments found. Creating new log.
I20250815 19:59:46.700345 25292 log.cc:826] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:46.705080 25292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c: No bootstrap required, opened a new log
I20250815 19:59:46.722249 25292 raft_consensus.cc:357] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46abfa7a5e394d71a5c55cfa384c532c" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39169 } }
I20250815 19:59:46.722903 25292 raft_consensus.cc:383] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:59:46.723099 25292 raft_consensus.cc:738] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46abfa7a5e394d71a5c55cfa384c532c, State: Initialized, Role: FOLLOWER
I20250815 19:59:46.723682 25292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46abfa7a5e394d71a5c55cfa384c532c" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39169 } }
I20250815 19:59:46.724143 25292 raft_consensus.cc:397] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 19:59:46.724368 25292 raft_consensus.cc:491] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 19:59:46.724633 25292 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:59:46.728490 25292 raft_consensus.cc:513] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46abfa7a5e394d71a5c55cfa384c532c" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39169 } }
I20250815 19:59:46.729153 25292 leader_election.cc:304] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46abfa7a5e394d71a5c55cfa384c532c; no voters:
I20250815 19:59:46.730795 25292 leader_election.cc:290] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 19:59:46.731722 25297 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:59:46.734139 25297 raft_consensus.cc:695] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [term 1 LEADER]: Becoming Leader. State: Replica: 46abfa7a5e394d71a5c55cfa384c532c, State: Running, Role: LEADER
I20250815 19:59:46.734719 25292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [sys.catalog]: configured and running, proceeding with master startup.
I20250815 19:59:46.734956 25297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46abfa7a5e394d71a5c55cfa384c532c" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39169 } }
I20250815 19:59:46.746768 25298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46abfa7a5e394d71a5c55cfa384c532c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46abfa7a5e394d71a5c55cfa384c532c" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39169 } } }
I20250815 19:59:46.747069 25299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46abfa7a5e394d71a5c55cfa384c532c. Latest consensus state: current_term: 1 leader_uuid: "46abfa7a5e394d71a5c55cfa384c532c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46abfa7a5e394d71a5c55cfa384c532c" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 39169 } } }
I20250815 19:59:46.747591 25298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:46.747898 25299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c [sys.catalog]: This master's current role is: LEADER
I20250815 19:59:46.750653 25307 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 19:59:46.763132 25307 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 19:59:46.781001 25307 catalog_manager.cc:1349] Generated new cluster ID: 2b8840a93d9a45a7a24cc21b808e982b
I20250815 19:59:46.781265 25307 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 19:59:46.801692 25307 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 19:59:46.803236 25307 catalog_manager.cc:1506] Loading token signing keys...
I20250815 19:59:46.826117 25307 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 46abfa7a5e394d71a5c55cfa384c532c: Generated new TSK 0
I20250815 19:59:46.827242 25307 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 19:59:46.841210 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--builtin_ntp_servers=127.20.53.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 19:59:47.160553 25316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:47.161104 25316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:47.161631 25316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:47.192574 25316 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 19:59:47.192988 25316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:47.193754 25316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 19:59:47.229386 25316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:47.230749 25316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:47.232339 25316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:47.244681 25322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:48.648643 25321 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 25316
W20250815 19:59:49.027194 25321 kernel_stack_watchdog.cc:198] Thread 25316 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 19:59:47.246357 25323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:49.029151 25316 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.784s user 0.704s sys 1.070s
W20250815 19:59:49.029443 25325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:49.029644 25316 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.784s user 0.704s sys 1.071s
W20250815 19:59:49.034977 25324 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1788 milliseconds
I20250815 19:59:49.035014 25316 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:49.036578 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:49.039253 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:49.040761 25316 hybrid_clock.cc:648] HybridClock initialized: now 1755287989040694 us; error 49 us; skew 500 ppm
I20250815 19:59:49.041834 25316 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:49.049275 25316 webserver.cc:480] Webserver started at http://127.20.53.65:40275/ using document root <none> and password file <none>
I20250815 19:59:49.050508 25316 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:49.050763 25316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:49.051390 25316 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:59:49.057868 25316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "432abfee994a4a4a9f40711c8ed790b0"
format_stamp: "Formatted at 2025-08-15 19:59:49 on dist-test-slave-d4bs"
I20250815 19:59:49.059350 25316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "432abfee994a4a4a9f40711c8ed790b0"
format_stamp: "Formatted at 2025-08-15 19:59:49 on dist-test-slave-d4bs"
I20250815 19:59:49.068591 25316 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20250815 19:59:49.074362 25332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:49.075537 25316 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250815 19:59:49.075848 25316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "432abfee994a4a4a9f40711c8ed790b0"
format_stamp: "Formatted at 2025-08-15 19:59:49 on dist-test-slave-d4bs"
I20250815 19:59:49.076202 25316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:49.150187 25316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:49.151724 25316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:49.152170 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:49.154749 25316 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:49.159421 25316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:59:49.159626 25316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:49.159853 25316 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:59:49.160012 25316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:49.333985 25316 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:34815
I20250815 19:59:49.334098 25444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:34815 every 8 connection(s)
I20250815 19:59:49.336838 25316 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 19:59:49.346101 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25316
I20250815 19:59:49.346589 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 19:59:49.354269 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--builtin_ntp_servers=127.20.53.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 19:59:49.366788 25445 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39169
I20250815 19:59:49.367316 25445 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:49.368625 25445 heartbeater.cc:507] Master 127.20.53.126:39169 requested a full tablet report, sending...
I20250815 19:59:49.371764 25257 ts_manager.cc:194] Registered new tserver with Master: 432abfee994a4a4a9f40711c8ed790b0 (127.20.53.65:34815)
I20250815 19:59:49.374738 25257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:49611
W20250815 19:59:49.677135 25449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:49.677855 25449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:49.678586 25449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:49.714425 25449 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 19:59:49.714780 25449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:49.715543 25449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 19:59:49.749428 25449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:49.750735 25449 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:49.752310 25449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:49.763712 25455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:50.378912 25445 heartbeater.cc:499] Master 127.20.53.126:39169 was elected leader, sending a full tablet report...
W20250815 19:59:49.765458 25456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:49.768698 25458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:50.917243 25457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 19:59:50.917363 25449 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:50.921557 25449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:50.923760 25449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:50.925117 25449 hybrid_clock.cc:648] HybridClock initialized: now 1755287990925081 us; error 43 us; skew 500 ppm
I20250815 19:59:50.925899 25449 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:50.931864 25449 webserver.cc:480] Webserver started at http://127.20.53.66:42949/ using document root <none> and password file <none>
I20250815 19:59:50.932860 25449 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:50.933080 25449 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:50.933543 25449 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:59:50.937990 25449 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c506480fb8bb4f45b9f500adb8e02e44"
format_stamp: "Formatted at 2025-08-15 19:59:50 on dist-test-slave-d4bs"
I20250815 19:59:50.939081 25449 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c506480fb8bb4f45b9f500adb8e02e44"
format_stamp: "Formatted at 2025-08-15 19:59:50 on dist-test-slave-d4bs"
I20250815 19:59:50.945983 25449 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250815 19:59:50.951637 25466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:50.952777 25449 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250815 19:59:50.953065 25449 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c506480fb8bb4f45b9f500adb8e02e44"
format_stamp: "Formatted at 2025-08-15 19:59:50 on dist-test-slave-d4bs"
I20250815 19:59:50.953364 25449 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:51.002856 25449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:51.004310 25449 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:51.004761 25449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:51.007272 25449 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:51.011454 25449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:59:51.011644 25449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:51.011873 25449 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:59:51.012027 25449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:51.144282 25449 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:41819
I20250815 19:59:51.144381 25578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:41819 every 8 connection(s)
I20250815 19:59:51.147081 25449 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 19:59:51.152799 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25449
I20250815 19:59:51.153517 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 19:59:51.159904 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--builtin_ntp_servers=127.20.53.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 19:59:51.169467 25579 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39169
I20250815 19:59:51.169894 25579 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:51.170857 25579 heartbeater.cc:507] Master 127.20.53.126:39169 requested a full tablet report, sending...
I20250815 19:59:51.173038 25257 ts_manager.cc:194] Registered new tserver with Master: c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819)
I20250815 19:59:51.174139 25257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:56997
W20250815 19:59:51.461717 25583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:51.462276 25583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:51.462783 25583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:51.493827 25583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 19:59:51.494258 25583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:51.495067 25583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 19:59:51.529233 25583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:51.530593 25583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:51.532176 25583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:51.543653 25589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:52.177011 25579 heartbeater.cc:499] Master 127.20.53.126:39169 was elected leader, sending a full tablet report...
W20250815 19:59:51.545390 25590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:52.783808 25592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:52.786144 25591 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1237 milliseconds
I20250815 19:59:52.786275 25583 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:52.787531 25583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:52.790313 25583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:52.791862 25583 hybrid_clock.cc:648] HybridClock initialized: now 1755287992791805 us; error 48 us; skew 500 ppm
I20250815 19:59:52.792810 25583 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:52.803894 25583 webserver.cc:480] Webserver started at http://127.20.53.67:39953/ using document root <none> and password file <none>
I20250815 19:59:52.804875 25583 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:52.805125 25583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:52.805608 25583 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:59:52.809842 25583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5bab16681bea43c0941c13d9f1ca313c"
format_stamp: "Formatted at 2025-08-15 19:59:52 on dist-test-slave-d4bs"
I20250815 19:59:52.811090 25583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5bab16681bea43c0941c13d9f1ca313c"
format_stamp: "Formatted at 2025-08-15 19:59:52 on dist-test-slave-d4bs"
I20250815 19:59:52.818161 25583 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250815 19:59:52.823712 25599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:52.824663 25583 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250815 19:59:52.825017 25583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5bab16681bea43c0941c13d9f1ca313c"
format_stamp: "Formatted at 2025-08-15 19:59:52 on dist-test-slave-d4bs"
I20250815 19:59:52.825352 25583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:52.904285 25583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:52.905815 25583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:52.906248 25583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:52.908690 25583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:52.912874 25583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:59:52.913084 25583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:52.913316 25583 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:59:52.913476 25583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:53.059325 25583 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:36221
I20250815 19:59:53.059439 25711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:36221 every 8 connection(s)
I20250815 19:59:53.062328 25583 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 19:59:53.065960 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25583
I20250815 19:59:53.066502 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 19:59:53.073048 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.68:0
--local_ip_for_outbound_sockets=127.20.53.68
--webserver_interface=127.20.53.68
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--builtin_ntp_servers=127.20.53.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 19:59:53.083021 25712 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39169
I20250815 19:59:53.083448 25712 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:53.084444 25712 heartbeater.cc:507] Master 127.20.53.126:39169 requested a full tablet report, sending...
I20250815 19:59:53.086678 25257 ts_manager.cc:194] Registered new tserver with Master: 5bab16681bea43c0941c13d9f1ca313c (127.20.53.67:36221)
I20250815 19:59:53.087980 25257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:57211
W20250815 19:59:53.377023 25716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:53.377532 25716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:53.378022 25716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:53.408598 25716 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 19:59:53.409015 25716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:53.409796 25716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.68
I20250815 19:59:53.444388 25716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.53.68
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.68
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:53.445704 25716 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:53.447278 25716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:53.458748 25722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:54.091122 25712 heartbeater.cc:499] Master 127.20.53.126:39169 was elected leader, sending a full tablet report...
W20250815 19:59:53.459666 25723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:54.711814 25725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:54.713912 25716 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.255s user 0.376s sys 0.870s
W20250815 19:59:54.714286 25716 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.255s user 0.377s sys 0.871s
W20250815 19:59:54.715132 25724 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1250 milliseconds
I20250815 19:59:54.715210 25716 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:54.716915 25716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:54.719705 25716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:54.721220 25716 hybrid_clock.cc:648] HybridClock initialized: now 1755287994721183 us; error 39 us; skew 500 ppm
I20250815 19:59:54.722443 25716 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:54.731456 25716 webserver.cc:480] Webserver started at http://127.20.53.68:46683/ using document root <none> and password file <none>
I20250815 19:59:54.732954 25716 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:54.733266 25716 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:54.733907 25716 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:59:54.741376 25716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2bb170c91bdc4652afacc65b0eb00856"
format_stamp: "Formatted at 2025-08-15 19:59:54 on dist-test-slave-d4bs"
I20250815 19:59:54.742997 25716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2bb170c91bdc4652afacc65b0eb00856"
format_stamp: "Formatted at 2025-08-15 19:59:54 on dist-test-slave-d4bs"
I20250815 19:59:54.753345 25716 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20250815 19:59:54.759586 25732 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:54.760928 25716 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.003s
I20250815 19:59:54.761258 25716 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2bb170c91bdc4652afacc65b0eb00856"
format_stamp: "Formatted at 2025-08-15 19:59:54 on dist-test-slave-d4bs"
I20250815 19:59:54.761588 25716 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:54.843940 25716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:54.845429 25716 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:54.845849 25716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:54.848387 25716 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:54.852941 25716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:59:54.853192 25716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:54.853443 25716 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:59:54.853603 25716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:54.985146 25716 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.68:44217
I20250815 19:59:54.985302 25844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.68:44217 every 8 connection(s)
I20250815 19:59:54.987782 25716 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 19:59:54.995992 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25716
I20250815 19:59:54.996531 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 19:59:55.004530 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.69:0
--local_ip_for_outbound_sockets=127.20.53.69
--webserver_interface=127.20.53.69
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--builtin_ntp_servers=127.20.53.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 19:59:55.017804 25845 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39169
I20250815 19:59:55.018285 25845 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:55.019423 25845 heartbeater.cc:507] Master 127.20.53.126:39169 requested a full tablet report, sending...
I20250815 19:59:55.021731 25257 ts_manager.cc:194] Registered new tserver with Master: 2bb170c91bdc4652afacc65b0eb00856 (127.20.53.68:44217)
I20250815 19:59:55.023001 25257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.68:49075
W20250815 19:59:55.319355 25849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 19:59:55.319869 25849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 19:59:55.320477 25849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 19:59:55.352890 25849 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 19:59:55.353324 25849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 19:59:55.354125 25849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.69
I20250815 19:59:55.390100 25849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.53.69
--webserver_port=0
--tserver_master_addrs=127.20.53.126:39169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.69
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 19:59:55.391412 25849 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 19:59:55.393033 25849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 19:59:55.406104 25856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 19:59:56.026403 25845 heartbeater.cc:499] Master 127.20.53.126:39169 was elected leader, sending a full tablet report...
W20250815 19:59:55.409781 25859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:55.406737 25857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 19:59:56.563308 25858 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 19:59:56.563445 25849 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 19:59:56.567325 25849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 19:59:56.570003 25849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 19:59:56.571434 25849 hybrid_clock.cc:648] HybridClock initialized: now 1755287996571395 us; error 41 us; skew 500 ppm
I20250815 19:59:56.572167 25849 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 19:59:56.578856 25849 webserver.cc:480] Webserver started at http://127.20.53.69:43119/ using document root <none> and password file <none>
I20250815 19:59:56.579738 25849 fs_manager.cc:362] Metadata directory not provided
I20250815 19:59:56.579933 25849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 19:59:56.580353 25849 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 19:59:56.585497 25849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d0ecad6538a14c348781b02a9c796632"
format_stamp: "Formatted at 2025-08-15 19:59:56 on dist-test-slave-d4bs"
I20250815 19:59:56.586723 25849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d0ecad6538a14c348781b02a9c796632"
format_stamp: "Formatted at 2025-08-15 19:59:56 on dist-test-slave-d4bs"
I20250815 19:59:56.593845 25849 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250815 19:59:56.599567 25866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:56.600579 25849 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250815 19:59:56.600930 25849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d0ecad6538a14c348781b02a9c796632"
format_stamp: "Formatted at 2025-08-15 19:59:56 on dist-test-slave-d4bs"
I20250815 19:59:56.601253 25849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 19:59:56.680313 25849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 19:59:56.681828 25849 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 19:59:56.682240 25849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 19:59:56.684695 25849 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 19:59:56.688769 25849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 19:59:56.688990 25849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 19:59:56.689251 25849 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 19:59:56.689407 25849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 19:59:56.843173 25849 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.69:43185
I20250815 19:59:56.843259 25978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.69:43185 every 8 connection(s)
I20250815 19:59:56.845803 25849 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250815 19:59:56.852496 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 25849
I20250815 19:59:56.852873 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250815 19:59:56.868033 25979 heartbeater.cc:344] Connected to a master server at 127.20.53.126:39169
I20250815 19:59:56.868456 25979 heartbeater.cc:461] Registering TS with master...
I20250815 19:59:56.869483 25979 heartbeater.cc:507] Master 127.20.53.126:39169 requested a full tablet report, sending...
I20250815 19:59:56.872529 25257 ts_manager.cc:194] Registered new tserver with Master: d0ecad6538a14c348781b02a9c796632 (127.20.53.69:43185)
I20250815 19:59:56.874331 20693 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 19:59:56.874276 25257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.69:54809
I20250815 19:59:56.913333 25257 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50210:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 19:59:57.008734 25514 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2ca93f7a914cd686503898116c0e4e (DEFAULT_TABLE table=TestTable [id=2e6b7cce38524bcca18aebe46c2224a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:59:57.010785 25514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2ca93f7a914cd686503898116c0e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:57.015048 25647 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2ca93f7a914cd686503898116c0e4e (DEFAULT_TABLE table=TestTable [id=2e6b7cce38524bcca18aebe46c2224a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:59:57.017820 25647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2ca93f7a914cd686503898116c0e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:57.017460 25780 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2ca93f7a914cd686503898116c0e4e (DEFAULT_TABLE table=TestTable [id=2e6b7cce38524bcca18aebe46c2224a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 19:59:57.019244 25780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2ca93f7a914cd686503898116c0e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 19:59:57.039819 25998 tablet_bootstrap.cc:492] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: Bootstrap starting.
I20250815 19:59:57.046911 25999 tablet_bootstrap.cc:492] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: Bootstrap starting.
I20250815 19:59:57.049402 25998 tablet_bootstrap.cc:654] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: Neither blocks nor log segments found. Creating new log.
I20250815 19:59:57.049669 26000 tablet_bootstrap.cc:492] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: Bootstrap starting.
I20250815 19:59:57.051487 25998 log.cc:826] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:57.055912 25999 tablet_bootstrap.cc:654] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: Neither blocks nor log segments found. Creating new log.
I20250815 19:59:57.057379 25998 tablet_bootstrap.cc:492] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: No bootstrap required, opened a new log
I20250815 19:59:57.057998 25998 ts_tablet_manager.cc:1397] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.012s
I20250815 19:59:57.058302 26000 tablet_bootstrap.cc:654] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: Neither blocks nor log segments found. Creating new log.
I20250815 19:59:57.060451 26000 log.cc:826] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:57.063874 25999 log.cc:826] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: Log is configured to *not* fsync() on all Append() calls
I20250815 19:59:57.066502 26000 tablet_bootstrap.cc:492] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: No bootstrap required, opened a new log
I20250815 19:59:57.067032 26000 ts_tablet_manager.cc:1397] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20250815 19:59:57.083011 25999 tablet_bootstrap.cc:492] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: No bootstrap required, opened a new log
I20250815 19:59:57.083849 25999 ts_tablet_manager.cc:1397] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.013s
I20250815 19:59:57.088266 25998 raft_consensus.cc:357] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.089541 25998 raft_consensus.cc:383] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:59:57.089890 25998 raft_consensus.cc:738] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c506480fb8bb4f45b9f500adb8e02e44, State: Initialized, Role: FOLLOWER
I20250815 19:59:57.090868 25998 consensus_queue.cc:260] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.094735 26000 raft_consensus.cc:357] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.095983 26000 raft_consensus.cc:383] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:59:57.096406 26000 raft_consensus.cc:738] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb170c91bdc4652afacc65b0eb00856, State: Initialized, Role: FOLLOWER
I20250815 19:59:57.097347 26000 consensus_queue.cc:260] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.098201 25998 ts_tablet_manager.cc:1428] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: Time spent starting tablet: real 0.040s user 0.023s sys 0.013s
I20250815 19:59:57.105875 26000 ts_tablet_manager.cc:1428] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: Time spent starting tablet: real 0.039s user 0.035s sys 0.001s
I20250815 19:59:57.114825 25999 raft_consensus.cc:357] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.115949 25999 raft_consensus.cc:383] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 19:59:57.116894 25999 raft_consensus.cc:738] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bab16681bea43c0941c13d9f1ca313c, State: Initialized, Role: FOLLOWER
I20250815 19:59:57.117966 25999 consensus_queue.cc:260] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.122025 25999 ts_tablet_manager.cc:1428] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: Time spent starting tablet: real 0.038s user 0.026s sys 0.008s
W20250815 19:59:57.158102 25580 tablet.cc:2378] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:59:57.215101 26005 raft_consensus.cc:491] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 19:59:57.215588 26005 raft_consensus.cc:513] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.218058 26005 leader_election.cc:290] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819), 5bab16681bea43c0941c13d9f1ca313c (127.20.53.67:36221)
I20250815 19:59:57.230005 25534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2ca93f7a914cd686503898116c0e4e" candidate_uuid: "2bb170c91bdc4652afacc65b0eb00856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c506480fb8bb4f45b9f500adb8e02e44" is_pre_election: true
I20250815 19:59:57.229987 25667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2ca93f7a914cd686503898116c0e4e" candidate_uuid: "2bb170c91bdc4652afacc65b0eb00856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bab16681bea43c0941c13d9f1ca313c" is_pre_election: true
I20250815 19:59:57.230737 25534 raft_consensus.cc:2466] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bb170c91bdc4652afacc65b0eb00856 in term 0.
I20250815 19:59:57.230763 25667 raft_consensus.cc:2466] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bb170c91bdc4652afacc65b0eb00856 in term 0.
I20250815 19:59:57.231964 25735 leader_election.cc:304] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb170c91bdc4652afacc65b0eb00856, 5bab16681bea43c0941c13d9f1ca313c; no voters:
I20250815 19:59:57.232744 26005 raft_consensus.cc:2802] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 19:59:57.233018 26005 raft_consensus.cc:491] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 19:59:57.233264 26005 raft_consensus.cc:3058] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:59:57.238173 26005 raft_consensus.cc:513] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.239653 26005 leader_election.cc:290] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [CANDIDATE]: Term 1 election: Requested vote from peers c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819), 5bab16681bea43c0941c13d9f1ca313c (127.20.53.67:36221)
I20250815 19:59:57.240406 25534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2ca93f7a914cd686503898116c0e4e" candidate_uuid: "2bb170c91bdc4652afacc65b0eb00856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c506480fb8bb4f45b9f500adb8e02e44"
I20250815 19:59:57.240636 25667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2ca93f7a914cd686503898116c0e4e" candidate_uuid: "2bb170c91bdc4652afacc65b0eb00856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bab16681bea43c0941c13d9f1ca313c"
I20250815 19:59:57.240971 25534 raft_consensus.cc:3058] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [term 0 FOLLOWER]: Advancing to term 1
I20250815 19:59:57.241168 25667 raft_consensus.cc:3058] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 0 FOLLOWER]: Advancing to term 1
W20250815 19:59:57.244653 25846 tablet.cc:2378] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:59:57.248062 25534 raft_consensus.cc:2466] T 5e2ca93f7a914cd686503898116c0e4e P c506480fb8bb4f45b9f500adb8e02e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb170c91bdc4652afacc65b0eb00856 in term 1.
I20250815 19:59:57.248062 25667 raft_consensus.cc:2466] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb170c91bdc4652afacc65b0eb00856 in term 1.
I20250815 19:59:57.249260 25733 leader_election.cc:304] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb170c91bdc4652afacc65b0eb00856, c506480fb8bb4f45b9f500adb8e02e44; no voters:
I20250815 19:59:57.249926 26005 raft_consensus.cc:2802] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 19:59:57.251974 26005 raft_consensus.cc:695] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [term 1 LEADER]: Becoming Leader. State: Replica: 2bb170c91bdc4652afacc65b0eb00856, State: Running, Role: LEADER
I20250815 19:59:57.252821 26005 consensus_queue.cc:237] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } }
I20250815 19:59:57.262629 25256 catalog_manager.cc:5582] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bb170c91bdc4652afacc65b0eb00856 (127.20.53.68). New cstate: current_term: 1 leader_uuid: "2bb170c91bdc4652afacc65b0eb00856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c506480fb8bb4f45b9f500adb8e02e44" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 41819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bb170c91bdc4652afacc65b0eb00856" member_type: VOTER last_known_addr { host: "127.20.53.68" port: 44217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250815 19:59:57.327214 20693 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
W20250815 19:59:57.329092 25713 tablet.cc:2378] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 19:59:57.330530 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c506480fb8bb4f45b9f500adb8e02e44 to finish bootstrapping
I20250815 19:59:57.345681 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5bab16681bea43c0941c13d9f1ca313c to finish bootstrapping
I20250815 19:59:57.356096 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2bb170c91bdc4652afacc65b0eb00856 to finish bootstrapping
I20250815 19:59:57.366827 20693 test_util.cc:276] Using random seed: -711998957
I20250815 19:59:57.393575 20693 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250815 19:59:57.394464 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25449
I20250815 19:59:57.432380 25667 raft_consensus.cc:1273] T 5e2ca93f7a914cd686503898116c0e4e P 5bab16681bea43c0941c13d9f1ca313c [term 1 FOLLOWER]: Refusing update from remote peer 2bb170c91bdc4652afacc65b0eb00856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250815 19:59:57.432679 25733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.66:41819: connect: Connection refused (error 111)
I20250815 19:59:57.434978 26010 consensus_queue.cc:1035] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bab16681bea43c0941c13d9f1ca313c" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 36221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 19:59:57.437837 25733 consensus_peers.cc:489] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 -> Peer c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819): Couldn't send request to peer c506480fb8bb4f45b9f500adb8e02e44. Status: Network error: Client connection negotiation failed: client connection to 127.20.53.66:41819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 19:59:57.454545 26020 mvcc.cc:204] Tried to move back new op lower bound from 7189659637464641536 to 7189659636754894848. Current Snapshot: MvccSnapshot[applied={T|T < 7189659637464641536}]
I20250815 19:59:57.459928 26023 mvcc.cc:204] Tried to move back new op lower bound from 7189659637464641536 to 7189659636754894848. Current Snapshot: MvccSnapshot[applied={T|T < 7189659637464641536}]
I20250815 19:59:57.878468 25979 heartbeater.cc:499] Master 127.20.53.126:39169 was elected leader, sending a full tablet report...
W20250815 19:59:59.565588 25733 consensus_peers.cc:489] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 -> Peer c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819): Couldn't send request to peer c506480fb8bb4f45b9f500adb8e02e44. Status: Network error: Client connection negotiation failed: client connection to 127.20.53.66:41819: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250815 20:00:00.197347 25780 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 20:00:00.216394 25380 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 20:00:00.250599 25647 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 20:00:00.255229 25914 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 20:00:01.981201 25733 consensus_peers.cc:489] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 -> Peer c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819): Couldn't send request to peer c506480fb8bb4f45b9f500adb8e02e44. Status: Network error: Client connection negotiation failed: client connection to 127.20.53.66:41819: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250815 20:00:01.992540 25780 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 20:00:02.025336 25380 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 20:00:02.045112 25914 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 20:00:02.078212 25647 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 20:00:02.506019 25733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.66:41819: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250815 20:00:04.317606 25733 consensus_peers.cc:489] T 5e2ca93f7a914cd686503898116c0e4e P 2bb170c91bdc4652afacc65b0eb00856 -> Peer c506480fb8bb4f45b9f500adb8e02e44 (127.20.53.66:41819): Couldn't send request to peer c506480fb8bb4f45b9f500adb8e02e44. Status: Network error: Client connection negotiation failed: client connection to 127.20.53.66:41819: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250815 20:00:04.507659 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25316
I20250815 20:00:04.533528 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25583
I20250815 20:00:04.567334 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25716
I20250815 20:00:04.607395 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25849
I20250815 20:00:04.632503 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 25224
2025-08-15T20:00:04Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19833 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19833 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250815 20:00:04.696081 20693 test_util.cc:276] Using random seed: -704669719
I20250815 20:00:04.700349 20693 ts_itest-base.cc:115] Starting cluster with:
I20250815 20:00:04.700524 20693 ts_itest-base.cc:116] --------------
I20250815 20:00:04.700640 20693 ts_itest-base.cc:117] 1 tablet servers
I20250815 20:00:04.700851 20693 ts_itest-base.cc:118] 1 replicas per TS
I20250815 20:00:04.700966 20693 ts_itest-base.cc:119] --------------
2025-08-15T20:00:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T20:00:04Z Disabled control of system clock
I20250815 20:00:04.747884 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:46325
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:46325 with env {}
W20250815 20:00:05.060070 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:05.060670 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:05.061106 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:05.093007 26121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 20:00:05.093330 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:05.093538 26121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 20:00:05.093737 26121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 20:00:05.130164 26121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:46325
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:05.131487 26121 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:05.133128 26121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:05.144526 26127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:05.145772 26128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:06.345603 26130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:06.348377 26129 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 20:00:06.348573 26121 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.204s user 0.491s sys 0.712s
W20250815 20:00:06.349018 26121 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.204s user 0.492s sys 0.712s
I20250815 20:00:06.349344 26121 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:06.351030 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:06.354607 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:06.356175 26121 hybrid_clock.cc:648] HybridClock initialized: now 1755288006356130 us; error 59 us; skew 500 ppm
I20250815 20:00:06.357167 26121 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:06.365801 26121 webserver.cc:480] Webserver started at http://127.20.53.126:44675/ using document root <none> and password file <none>
I20250815 20:00:06.366837 26121 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:06.367084 26121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:06.367569 26121 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:06.372298 26121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "10093522d44b4a269b73e2b807590d4d"
format_stamp: "Formatted at 2025-08-15 20:00:06 on dist-test-slave-d4bs"
I20250815 20:00:06.373564 26121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "10093522d44b4a269b73e2b807590d4d"
format_stamp: "Formatted at 2025-08-15 20:00:06 on dist-test-slave-d4bs"
I20250815 20:00:06.382239 26121 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250815 20:00:06.388777 26137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:06.390112 26121 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 20:00:06.390465 26121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
uuid: "10093522d44b4a269b73e2b807590d4d"
format_stamp: "Formatted at 2025-08-15 20:00:06 on dist-test-slave-d4bs"
I20250815 20:00:06.390808 26121 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:06.462450 26121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:06.463915 26121 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:06.464366 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:06.535228 26121 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:46325
I20250815 20:00:06.535296 26188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:46325 every 8 connection(s)
I20250815 20:00:06.537987 26121 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 20:00:06.540390 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 26121
I20250815 20:00:06.541137 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 20:00:06.543709 26189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:06.567531 26189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d: Bootstrap starting.
I20250815 20:00:06.573698 26189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:06.575390 26189 log.cc:826] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:06.579556 26189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d: No bootstrap required, opened a new log
I20250815 20:00:06.597345 26189 raft_consensus.cc:357] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10093522d44b4a269b73e2b807590d4d" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 46325 } }
I20250815 20:00:06.598062 26189 raft_consensus.cc:383] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:06.598341 26189 raft_consensus.cc:738] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10093522d44b4a269b73e2b807590d4d, State: Initialized, Role: FOLLOWER
I20250815 20:00:06.599041 26189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10093522d44b4a269b73e2b807590d4d" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 46325 } }
I20250815 20:00:06.599538 26189 raft_consensus.cc:397] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 20:00:06.599802 26189 raft_consensus.cc:491] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 20:00:06.600095 26189 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:06.604157 26189 raft_consensus.cc:513] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10093522d44b4a269b73e2b807590d4d" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 46325 } }
I20250815 20:00:06.604913 26189 leader_election.cc:304] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10093522d44b4a269b73e2b807590d4d; no voters:
I20250815 20:00:06.606796 26189 leader_election.cc:290] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 20:00:06.607569 26194 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 1 FOLLOWER]: Leader election won for term 1
I20250815 20:00:06.609954 26194 raft_consensus.cc:695] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [term 1 LEADER]: Becoming Leader. State: Replica: 10093522d44b4a269b73e2b807590d4d, State: Running, Role: LEADER
I20250815 20:00:06.610845 26194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10093522d44b4a269b73e2b807590d4d" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 46325 } }
I20250815 20:00:06.611678 26189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [sys.catalog]: configured and running, proceeding with master startup.
I20250815 20:00:06.617626 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10093522d44b4a269b73e2b807590d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10093522d44b4a269b73e2b807590d4d" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 46325 } } }
I20250815 20:00:06.618309 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:06.619853 26196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10093522d44b4a269b73e2b807590d4d. Latest consensus state: current_term: 1 leader_uuid: "10093522d44b4a269b73e2b807590d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10093522d44b4a269b73e2b807590d4d" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 46325 } } }
I20250815 20:00:06.621572 26196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:06.626472 26201 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 20:00:06.640785 26201 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 20:00:06.658171 26201 catalog_manager.cc:1349] Generated new cluster ID: bd77553186f14ea99920ea29481700f8
I20250815 20:00:06.658526 26201 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 20:00:06.683333 26201 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 20:00:06.684872 26201 catalog_manager.cc:1506] Loading token signing keys...
I20250815 20:00:06.699879 26201 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 10093522d44b4a269b73e2b807590d4d: Generated new TSK 0
I20250815 20:00:06.700861 26201 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 20:00:06.711663 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:46325
--builtin_ntp_servers=127.20.53.84:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 20:00:07.034126 26213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:07.034650 26213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:07.035158 26213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:07.067420 26213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:07.068459 26213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 20:00:07.103794 26213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:46325
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:07.105199 26213 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:07.106791 26213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:07.120199 26219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:07.120438 26220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:08.521201 26218 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 26213
W20250815 20:00:08.645138 26213 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.527s user 0.002s sys 0.000s
W20250815 20:00:08.645478 26213 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.527s user 0.002s sys 0.000s
I20250815 20:00:08.654855 26213 server_base.cc:1047] running on GCE node
W20250815 20:00:08.654886 26223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:08.656101 26213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:08.658643 26213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:08.660089 26213 hybrid_clock.cc:648] HybridClock initialized: now 1755288008660043 us; error 51 us; skew 500 ppm
I20250815 20:00:08.660871 26213 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:08.666663 26213 webserver.cc:480] Webserver started at http://127.20.53.65:41829/ using document root <none> and password file <none>
I20250815 20:00:08.667557 26213 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:08.667744 26213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:08.668136 26213 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:08.672513 26213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0b78f1c7638b43818291d6e8511522d0"
format_stamp: "Formatted at 2025-08-15 20:00:08 on dist-test-slave-d4bs"
I20250815 20:00:08.673595 26213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0b78f1c7638b43818291d6e8511522d0"
format_stamp: "Formatted at 2025-08-15 20:00:08 on dist-test-slave-d4bs"
I20250815 20:00:08.680931 26213 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250815 20:00:08.686475 26229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:08.687503 26213 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 20:00:08.687826 26213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0b78f1c7638b43818291d6e8511522d0"
format_stamp: "Formatted at 2025-08-15 20:00:08 on dist-test-slave-d4bs"
I20250815 20:00:08.688170 26213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:08.750830 26213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:08.752410 26213 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:08.752875 26213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:08.755390 26213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:08.759490 26213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 20:00:08.759718 26213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:08.759974 26213 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 20:00:08.760154 26213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:08.892659 26213 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:44547
I20250815 20:00:08.892805 26341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:44547 every 8 connection(s)
I20250815 20:00:08.895256 26213 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 20:00:08.905740 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 26213
I20250815 20:00:08.906160 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755287859650099-20693-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 20:00:08.915891 26342 heartbeater.cc:344] Connected to a master server at 127.20.53.126:46325
I20250815 20:00:08.916327 26342 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:08.917407 26342 heartbeater.cc:507] Master 127.20.53.126:46325 requested a full tablet report, sending...
I20250815 20:00:08.919726 26154 ts_manager.cc:194] Registered new tserver with Master: 0b78f1c7638b43818291d6e8511522d0 (127.20.53.65:44547)
I20250815 20:00:08.921694 26154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:59745
I20250815 20:00:08.925444 20693 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 20:00:08.956339 26154 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41982:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 20:00:09.012231 26277 tablet_service.cc:1468] Processing CreateTablet for tablet f7644114f1cc4ea49c95d13808c12650 (DEFAULT_TABLE table=TestTable [id=59ab99d03cd643d2969e383704d9b9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:09.013993 26277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7644114f1cc4ea49c95d13808c12650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:09.033860 26357 tablet_bootstrap.cc:492] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0: Bootstrap starting.
I20250815 20:00:09.039934 26357 tablet_bootstrap.cc:654] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:09.041781 26357 log.cc:826] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:09.046667 26357 tablet_bootstrap.cc:492] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0: No bootstrap required, opened a new log
I20250815 20:00:09.047091 26357 ts_tablet_manager.cc:1397] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20250815 20:00:09.064394 26357 raft_consensus.cc:357] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b78f1c7638b43818291d6e8511522d0" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44547 } }
I20250815 20:00:09.065030 26357 raft_consensus.cc:383] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:09.065268 26357 raft_consensus.cc:738] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b78f1c7638b43818291d6e8511522d0, State: Initialized, Role: FOLLOWER
I20250815 20:00:09.065927 26357 consensus_queue.cc:260] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b78f1c7638b43818291d6e8511522d0" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44547 } }
I20250815 20:00:09.066434 26357 raft_consensus.cc:397] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 20:00:09.066699 26357 raft_consensus.cc:491] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 20:00:09.067041 26357 raft_consensus.cc:3058] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:09.071810 26357 raft_consensus.cc:513] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b78f1c7638b43818291d6e8511522d0" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44547 } }
I20250815 20:00:09.072525 26357 leader_election.cc:304] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b78f1c7638b43818291d6e8511522d0; no voters:
I20250815 20:00:09.074254 26357 leader_election.cc:290] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 20:00:09.074668 26359 raft_consensus.cc:2802] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 20:00:09.077234 26359 raft_consensus.cc:695] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0b78f1c7638b43818291d6e8511522d0, State: Running, Role: LEADER
I20250815 20:00:09.078153 26342 heartbeater.cc:499] Master 127.20.53.126:46325 was elected leader, sending a full tablet report...
I20250815 20:00:09.078241 26359 consensus_queue.cc:237] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b78f1c7638b43818291d6e8511522d0" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44547 } }
I20250815 20:00:09.078701 26357 ts_tablet_manager.cc:1428] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0: Time spent starting tablet: real 0.031s user 0.033s sys 0.000s
I20250815 20:00:09.090533 26153 catalog_manager.cc:5582] T f7644114f1cc4ea49c95d13808c12650 P 0b78f1c7638b43818291d6e8511522d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b78f1c7638b43818291d6e8511522d0 (127.20.53.65). New cstate: current_term: 1 leader_uuid: "0b78f1c7638b43818291d6e8511522d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b78f1c7638b43818291d6e8511522d0" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250815 20:00:09.116852 20693 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 20:00:09.119736 20693 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0b78f1c7638b43818291d6e8511522d0 to finish bootstrapping
I20250815 20:00:11.870360 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 26213
I20250815 20:00:11.896867 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 26121
2025-08-15T20:00:11Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7262 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7263 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250815 20:00:11.959805 20693 test_util.cc:276] Using random seed: -697405985
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-15T20:00:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T20:00:11Z Disabled control of system clock
I20250815 20:00:12.016786 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:41439
--webserver_interface=127.20.53.126
--webserver_port=0
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:41439 with env {}
W20250815 20:00:12.327432 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:12.328058 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:12.328537 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:12.360528 26386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 20:00:12.360899 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:12.361186 26386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 20:00:12.361404 26386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 20:00:12.397652 26386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:41439
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:12.398945 26386 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:12.400513 26386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:12.411370 26392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:12.411953 26393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:13.636580 26394 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1223 milliseconds
W20250815 20:00:13.636890 26386 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.225s user 0.484s sys 0.737s
W20250815 20:00:13.637334 26386 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.226s user 0.484s sys 0.737s
W20250815 20:00:13.637462 26395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:13.637662 26386 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:13.639333 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:13.642917 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:13.644454 26386 hybrid_clock.cc:648] HybridClock initialized: now 1755288013644408 us; error 39 us; skew 500 ppm
I20250815 20:00:13.645723 26386 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:13.658188 26386 webserver.cc:480] Webserver started at http://127.20.53.126:40321/ using document root <none> and password file <none>
I20250815 20:00:13.659624 26386 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:13.659948 26386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:13.660631 26386 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:13.668151 26386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/instance:
uuid: "4bace0e8ca59485d9eb83f464d19087a"
format_stamp: "Formatted at 2025-08-15 20:00:13 on dist-test-slave-d4bs"
I20250815 20:00:13.669894 26386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal/instance:
uuid: "4bace0e8ca59485d9eb83f464d19087a"
format_stamp: "Formatted at 2025-08-15 20:00:13 on dist-test-slave-d4bs"
I20250815 20:00:13.680667 26386 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20250815 20:00:13.688947 26402 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:13.690382 26386 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250815 20:00:13.690860 26386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
uuid: "4bace0e8ca59485d9eb83f464d19087a"
format_stamp: "Formatted at 2025-08-15 20:00:13 on dist-test-slave-d4bs"
I20250815 20:00:13.691372 26386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:13.778754 26386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:13.780992 26386 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:13.781657 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:13.853087 26386 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:41439
I20250815 20:00:13.853153 26453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:41439 every 8 connection(s)
I20250815 20:00:13.855861 26386 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb
I20250815 20:00:13.861266 26454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:13.865777 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 26386
I20250815 20:00:13.866220 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal/instance
I20250815 20:00:13.885211 26454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a: Bootstrap starting.
I20250815 20:00:13.891525 26454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:13.893347 26454 log.cc:826] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:13.897946 26454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a: No bootstrap required, opened a new log
I20250815 20:00:13.915568 26454 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bace0e8ca59485d9eb83f464d19087a" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:13.916288 26454 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:13.916524 26454 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bace0e8ca59485d9eb83f464d19087a, State: Initialized, Role: FOLLOWER
I20250815 20:00:13.917280 26454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bace0e8ca59485d9eb83f464d19087a" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:13.917788 26454 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 20:00:13.918042 26454 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 20:00:13.918349 26454 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:13.922827 26454 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bace0e8ca59485d9eb83f464d19087a" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:13.923552 26454 leader_election.cc:304] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bace0e8ca59485d9eb83f464d19087a; no voters:
I20250815 20:00:13.925228 26454 leader_election.cc:290] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 20:00:13.925971 26459 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 1 FOLLOWER]: Leader election won for term 1
I20250815 20:00:13.928130 26459 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [term 1 LEADER]: Becoming Leader. State: Replica: 4bace0e8ca59485d9eb83f464d19087a, State: Running, Role: LEADER
I20250815 20:00:13.929119 26459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bace0e8ca59485d9eb83f464d19087a" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:13.931010 26454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [sys.catalog]: configured and running, proceeding with master startup.
I20250815 20:00:13.936110 26460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bace0e8ca59485d9eb83f464d19087a. Latest consensus state: current_term: 1 leader_uuid: "4bace0e8ca59485d9eb83f464d19087a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bace0e8ca59485d9eb83f464d19087a" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } } }
I20250815 20:00:13.936342 26461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bace0e8ca59485d9eb83f464d19087a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bace0e8ca59485d9eb83f464d19087a" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } } }
I20250815 20:00:13.936899 26460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:13.937116 26461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:13.940173 26466 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 20:00:13.951896 26466 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 20:00:13.970005 26466 catalog_manager.cc:1349] Generated new cluster ID: 6137da0580954a2d96039aa0cf2ec46a
I20250815 20:00:13.970281 26466 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 20:00:13.996022 26466 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 20:00:13.997678 26466 catalog_manager.cc:1506] Loading token signing keys...
I20250815 20:00:14.015841 26466 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4bace0e8ca59485d9eb83f464d19087a: Generated new TSK 0
I20250815 20:00:14.016862 26466 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 20:00:14.035481 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:0
--local_ip_for_outbound_sockets=127.20.53.65
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:41439
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250815 20:00:14.355203 26478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:14.355737 26478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:14.356254 26478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:14.389509 26478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:14.390393 26478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 20:00:14.427073 26478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=0
--tserver_master_addrs=127.20.53.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:14.428543 26478 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:14.430469 26478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:14.444346 26484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:14.445467 26485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:15.661638 26487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:15.663246 26478 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.218s user 0.505s sys 0.710s
W20250815 20:00:15.663689 26478 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.219s user 0.507s sys 0.711s
I20250815 20:00:15.666661 26478 server_base.cc:1047] running on GCE node
I20250815 20:00:15.668242 26478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:15.671319 26478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:15.672868 26478 hybrid_clock.cc:648] HybridClock initialized: now 1755288015672769 us; error 101 us; skew 500 ppm
I20250815 20:00:15.674145 26478 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:15.683770 26478 webserver.cc:480] Webserver started at http://127.20.53.65:40929/ using document root <none> and password file <none>
I20250815 20:00:15.685292 26478 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:15.685613 26478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:15.686281 26478 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:15.693825 26478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/instance:
uuid: "5e7568cc9ace4a7093aa46eada6f8cba"
format_stamp: "Formatted at 2025-08-15 20:00:15 on dist-test-slave-d4bs"
I20250815 20:00:15.695555 26478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal/instance:
uuid: "5e7568cc9ace4a7093aa46eada6f8cba"
format_stamp: "Formatted at 2025-08-15 20:00:15 on dist-test-slave-d4bs"
I20250815 20:00:15.706797 26478 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.010s sys 0.003s
I20250815 20:00:15.715550 26494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:15.717024 26478 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250815 20:00:15.717495 26478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
uuid: "5e7568cc9ace4a7093aa46eada6f8cba"
format_stamp: "Formatted at 2025-08-15 20:00:15 on dist-test-slave-d4bs"
I20250815 20:00:15.717973 26478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:15.811089 26478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:15.812580 26478 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:15.813068 26478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:15.815722 26478 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:15.819975 26478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 20:00:15.820207 26478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:15.820453 26478 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 20:00:15.820613 26478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:15.987318 26478 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:33315
I20250815 20:00:15.987493 26606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:33315 every 8 connection(s)
I20250815 20:00:15.990191 26478 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb
I20250815 20:00:15.991456 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 26478
I20250815 20:00:15.991955 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal/instance
I20250815 20:00:16.001422 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:0
--local_ip_for_outbound_sockets=127.20.53.66
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:41439
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 20:00:16.028457 26607 heartbeater.cc:344] Connected to a master server at 127.20.53.126:41439
I20250815 20:00:16.028931 26607 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:16.029968 26607 heartbeater.cc:507] Master 127.20.53.126:41439 requested a full tablet report, sending...
I20250815 20:00:16.032482 26419 ts_manager.cc:194] Registered new tserver with Master: 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65:33315)
I20250815 20:00:16.034582 26419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:38491
W20250815 20:00:16.365419 26611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:16.365929 26611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:16.366453 26611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:16.398492 26611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:16.399391 26611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 20:00:16.434914 26611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=0
--tserver_master_addrs=127.20.53.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:16.436280 26611 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:16.438078 26611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:16.451290 26617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:17.038236 26607 heartbeater.cc:499] Master 127.20.53.126:41439 was elected leader, sending a full tablet report...
W20250815 20:00:17.861069 26616 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 26611
W20250815 20:00:18.216730 26611 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.765s user 0.694s sys 1.067s
W20250815 20:00:18.217180 26611 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.766s user 0.695s sys 1.067s
W20250815 20:00:16.453217 26618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:18.218423 26619 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1765 milliseconds
W20250815 20:00:18.219200 26620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:18.219177 26611 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:18.222509 26611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:18.224612 26611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:18.226029 26611 hybrid_clock.cc:648] HybridClock initialized: now 1755288018225985 us; error 53 us; skew 500 ppm
I20250815 20:00:18.226837 26611 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:18.234189 26611 webserver.cc:480] Webserver started at http://127.20.53.66:38107/ using document root <none> and password file <none>
I20250815 20:00:18.235364 26611 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:18.235591 26611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:18.236060 26611 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:18.240603 26611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/instance:
uuid: "f09629bae44a4fafbfcc26bc20a4d390"
format_stamp: "Formatted at 2025-08-15 20:00:18 on dist-test-slave-d4bs"
I20250815 20:00:18.241853 26611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal/instance:
uuid: "f09629bae44a4fafbfcc26bc20a4d390"
format_stamp: "Formatted at 2025-08-15 20:00:18 on dist-test-slave-d4bs"
I20250815 20:00:18.249523 26611 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 20:00:18.255692 26627 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:18.256858 26611 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 20:00:18.257238 26611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
uuid: "f09629bae44a4fafbfcc26bc20a4d390"
format_stamp: "Formatted at 2025-08-15 20:00:18 on dist-test-slave-d4bs"
I20250815 20:00:18.257608 26611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:18.326279 26611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:18.327811 26611 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:18.328289 26611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:18.330868 26611 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:18.335312 26611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 20:00:18.335546 26611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:18.335804 26611 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 20:00:18.335975 26611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:18.473532 26611 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:39779
I20250815 20:00:18.473627 26739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:39779 every 8 connection(s)
I20250815 20:00:18.476243 26611 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb
I20250815 20:00:18.483342 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 26611
I20250815 20:00:18.483826 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal/instance
I20250815 20:00:18.490804 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:0
--local_ip_for_outbound_sockets=127.20.53.67
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:41439
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 20:00:18.504170 26740 heartbeater.cc:344] Connected to a master server at 127.20.53.126:41439
I20250815 20:00:18.504655 26740 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:18.505821 26740 heartbeater.cc:507] Master 127.20.53.126:41439 requested a full tablet report, sending...
I20250815 20:00:18.508189 26419 ts_manager.cc:194] Registered new tserver with Master: f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779)
I20250815 20:00:18.509552 26419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:33183
W20250815 20:00:18.805122 26744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:18.805672 26744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:18.806193 26744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:18.838796 26744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:18.839679 26744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 20:00:18.875330 26744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=0
--tserver_master_addrs=127.20.53.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:18.876684 26744 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:18.878376 26744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:18.891005 26750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:19.514271 26740 heartbeater.cc:499] Master 127.20.53.126:41439 was elected leader, sending a full tablet report...
W20250815 20:00:18.891479 26751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:20.235410 26752 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1342 milliseconds
W20250815 20:00:20.236286 26753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:20.238787 26744 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.348s user 0.477s sys 0.863s
W20250815 20:00:20.239221 26744 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.349s user 0.479s sys 0.864s
I20250815 20:00:20.239557 26744 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:20.241472 26744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:20.244971 26744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:20.246531 26744 hybrid_clock.cc:648] HybridClock initialized: now 1755288020246485 us; error 47 us; skew 500 ppm
I20250815 20:00:20.247776 26744 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:20.257318 26744 webserver.cc:480] Webserver started at http://127.20.53.67:34995/ using document root <none> and password file <none>
I20250815 20:00:20.258867 26744 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:20.259213 26744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:20.259887 26744 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:20.267519 26744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/instance:
uuid: "e52de05298b44607b036b96f02be2fc5"
format_stamp: "Formatted at 2025-08-15 20:00:20 on dist-test-slave-d4bs"
I20250815 20:00:20.269373 26744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal/instance:
uuid: "e52de05298b44607b036b96f02be2fc5"
format_stamp: "Formatted at 2025-08-15 20:00:20 on dist-test-slave-d4bs"
I20250815 20:00:20.280176 26744 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.001s
I20250815 20:00:20.288600 26761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:20.290047 26744 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250815 20:00:20.290556 26744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
uuid: "e52de05298b44607b036b96f02be2fc5"
format_stamp: "Formatted at 2025-08-15 20:00:20 on dist-test-slave-d4bs"
I20250815 20:00:20.291096 26744 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:20.386663 26744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:20.388125 26744 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:20.388535 26744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:20.391073 26744 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:20.395316 26744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 20:00:20.395526 26744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:20.395731 26744 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 20:00:20.395866 26744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:20.530853 26744 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:38931
I20250815 20:00:20.530992 26873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:38931 every 8 connection(s)
I20250815 20:00:20.533605 26744 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb
I20250815 20:00:20.538264 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 26744
I20250815 20:00:20.538923 20693 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal/instance
I20250815 20:00:20.555931 26874 heartbeater.cc:344] Connected to a master server at 127.20.53.126:41439
I20250815 20:00:20.556406 26874 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:20.557828 26874 heartbeater.cc:507] Master 127.20.53.126:41439 requested a full tablet report, sending...
I20250815 20:00:20.560472 26419 ts_manager.cc:194] Registered new tserver with Master: e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:20.561914 26419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:49615
I20250815 20:00:20.575639 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 20:00:20.606168 20693 test_util.cc:276] Using random seed: -688759615
I20250815 20:00:20.646584 26419 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59924:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 20:00:20.649067 26419 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 20:00:20.713840 26542 tablet_service.cc:1468] Processing CreateTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 (DEFAULT_TABLE table=pre_rebuild [id=c6b99fdda52942778d932f9e3faba071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:20.714450 26675 tablet_service.cc:1468] Processing CreateTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 (DEFAULT_TABLE table=pre_rebuild [id=c6b99fdda52942778d932f9e3faba071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:20.715922 26542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d7b8929d68b4d5d93d86a838c87d857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:20.716380 26675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d7b8929d68b4d5d93d86a838c87d857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:20.715819 26809 tablet_service.cc:1468] Processing CreateTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 (DEFAULT_TABLE table=pre_rebuild [id=c6b99fdda52942778d932f9e3faba071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:20.717773 26809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d7b8929d68b4d5d93d86a838c87d857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:20.742697 26898 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Bootstrap starting.
I20250815 20:00:20.745960 26899 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Bootstrap starting.
I20250815 20:00:20.748077 26900 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Bootstrap starting.
I20250815 20:00:20.751045 26898 tablet_bootstrap.cc:654] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:20.753720 26898 log.cc:826] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:20.753993 26900 tablet_bootstrap.cc:654] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:20.756405 26899 tablet_bootstrap.cc:654] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:20.756325 26900 log.cc:826] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:20.759013 26899 log.cc:826] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:20.760078 26898 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: No bootstrap required, opened a new log
I20250815 20:00:20.760684 26898 ts_tablet_manager.cc:1397] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20250815 20:00:20.762601 26900 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: No bootstrap required, opened a new log
I20250815 20:00:20.763043 26900 ts_tablet_manager.cc:1397] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20250815 20:00:20.765028 26899 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: No bootstrap required, opened a new log
I20250815 20:00:20.765573 26899 ts_tablet_manager.cc:1397] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.009s
I20250815 20:00:20.781752 26900 raft_consensus.cc:357] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.782588 26900 raft_consensus.cc:383] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:20.782862 26900 raft_consensus.cc:738] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09629bae44a4fafbfcc26bc20a4d390, State: Initialized, Role: FOLLOWER
I20250815 20:00:20.783720 26900 consensus_queue.cc:260] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.789597 26900 ts_tablet_manager.cc:1428] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Time spent starting tablet: real 0.026s user 0.020s sys 0.007s
I20250815 20:00:20.789459 26898 raft_consensus.cc:357] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.790472 26898 raft_consensus.cc:383] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:20.790789 26898 raft_consensus.cc:738] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52de05298b44607b036b96f02be2fc5, State: Initialized, Role: FOLLOWER
I20250815 20:00:20.791743 26898 consensus_queue.cc:260] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.794718 26874 heartbeater.cc:499] Master 127.20.53.126:41439 was elected leader, sending a full tablet report...
I20250815 20:00:20.795744 26898 ts_tablet_manager.cc:1428] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Time spent starting tablet: real 0.034s user 0.027s sys 0.005s
I20250815 20:00:20.800226 26899 raft_consensus.cc:357] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.801309 26899 raft_consensus.cc:383] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:20.801631 26899 raft_consensus.cc:738] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7568cc9ace4a7093aa46eada6f8cba, State: Initialized, Role: FOLLOWER
I20250815 20:00:20.802572 26899 consensus_queue.cc:260] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.808446 26899 ts_tablet_manager.cc:1428] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Time spent starting tablet: real 0.043s user 0.029s sys 0.006s
I20250815 20:00:20.876446 26906 raft_consensus.cc:491] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 20:00:20.877089 26906 raft_consensus.cc:513] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.880303 26906 leader_election.cc:290] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:20.894949 26695 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390" is_pre_election: true
I20250815 20:00:20.895870 26695 raft_consensus.cc:2466] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 0.
I20250815 20:00:20.895924 26829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52de05298b44607b036b96f02be2fc5" is_pre_election: true
I20250815 20:00:20.896593 26829 raft_consensus.cc:2466] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 0.
I20250815 20:00:20.897323 26497 leader_election.cc:304] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba, f09629bae44a4fafbfcc26bc20a4d390; no voters:
I20250815 20:00:20.898211 26906 raft_consensus.cc:2802] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 20:00:20.898521 26906 raft_consensus.cc:491] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 20:00:20.898799 26906 raft_consensus.cc:3058] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:20.903564 26906 raft_consensus.cc:513] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.905372 26906 leader_election.cc:290] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 election: Requested vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:20.906324 26695 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390"
I20250815 20:00:20.906497 26829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52de05298b44607b036b96f02be2fc5"
I20250815 20:00:20.906762 26695 raft_consensus.cc:3058] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:20.907016 26829 raft_consensus.cc:3058] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:20.911559 26695 raft_consensus.cc:2466] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 1.
I20250815 20:00:20.911893 26829 raft_consensus.cc:2466] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 1.
I20250815 20:00:20.912544 26497 leader_election.cc:304] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba, f09629bae44a4fafbfcc26bc20a4d390; no voters:
I20250815 20:00:20.913290 26906 raft_consensus.cc:2802] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Leader election won for term 1
I20250815 20:00:20.914908 26906 raft_consensus.cc:695] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 LEADER]: Becoming Leader. State: Replica: 5e7568cc9ace4a7093aa46eada6f8cba, State: Running, Role: LEADER
I20250815 20:00:20.915827 26906 consensus_queue.cc:237] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:20.925345 26418 catalog_manager.cc:5582] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65). New cstate: current_term: 1 leader_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } health_report { overall_health: UNKNOWN } } }
W20250815 20:00:20.983897 26741 tablet.cc:2378] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 20:00:21.020992 26608 tablet.cc:2378] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 20:00:21.038584 26875 tablet.cc:2378] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 20:00:21.117894 26695 raft_consensus.cc:1273] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Refusing update from remote peer 5e7568cc9ace4a7093aa46eada6f8cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 20:00:21.117908 26829 raft_consensus.cc:1273] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Refusing update from remote peer 5e7568cc9ace4a7093aa46eada6f8cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 20:00:21.119544 26906 consensus_queue.cc:1035] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 20:00:21.120461 26909 consensus_queue.cc:1035] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 20:00:21.151400 26916 mvcc.cc:204] Tried to move back new op lower bound from 7189659734481641472 to 7189659733679083520. Current Snapshot: MvccSnapshot[applied={T|T < 7189659734481641472}]
I20250815 20:00:21.156649 26918 mvcc.cc:204] Tried to move back new op lower bound from 7189659734481641472 to 7189659733679083520. Current Snapshot: MvccSnapshot[applied={T|T < 7189659734481641472}]
I20250815 20:00:21.158360 26919 mvcc.cc:204] Tried to move back new op lower bound from 7189659734481641472 to 7189659733679083520. Current Snapshot: MvccSnapshot[applied={T|T < 7189659734481641472}]
I20250815 20:00:26.496188 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 26386
W20250815 20:00:26.883096 26952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:26.883735 26952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:26.916575 26952 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 20:00:27.231848 26874 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:41439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:41439: connect: Connection refused (error 111)
W20250815 20:00:27.267210 26607 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:41439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:41439: connect: Connection refused (error 111)
W20250815 20:00:27.277367 26740 heartbeater.cc:646] Failed to heartbeat to 127.20.53.126:41439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.53.126:41439: connect: Connection refused (error 111)
W20250815 20:00:28.364929 26957 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 26952
W20250815 20:00:28.365515 26957 kernel_stack_watchdog.cc:198] Thread 26952 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 20:00:28.543133 26952 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.583s user 0.589s sys 0.985s
W20250815 20:00:28.647040 26952 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.687s user 0.591s sys 0.996s
I20250815 20:00:28.731292 26952 minidump.cc:252] Setting minidump size limit to 20M
I20250815 20:00:28.733368 26952 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:28.734534 26952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:28.745291 26986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:28.746264 26987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:28.749435 26989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:28.749578 26952 server_base.cc:1047] running on GCE node
I20250815 20:00:28.750764 26952 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250815 20:00:28.751273 26952 hybrid_clock.cc:648] HybridClock initialized: now 1755288028751246 us; error 148073 us; skew 500 ppm
I20250815 20:00:28.752005 26952 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:28.756749 26952 webserver.cc:480] Webserver started at http://0.0.0.0:43519/ using document root <none> and password file <none>
I20250815 20:00:28.757679 26952 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:28.757925 26952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:28.758385 26952 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 20:00:28.762852 26952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/instance:
uuid: "d563201e36f8477684240c2122f7fa26"
format_stamp: "Formatted at 2025-08-15 20:00:28 on dist-test-slave-d4bs"
I20250815 20:00:28.764081 26952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal/instance:
uuid: "d563201e36f8477684240c2122f7fa26"
format_stamp: "Formatted at 2025-08-15 20:00:28 on dist-test-slave-d4bs"
I20250815 20:00:28.770795 26952 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250815 20:00:28.775787 26994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:28.776808 26952 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250815 20:00:28.777158 26952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
uuid: "d563201e36f8477684240c2122f7fa26"
format_stamp: "Formatted at 2025-08-15 20:00:28 on dist-test-slave-d4bs"
I20250815 20:00:28.777505 26952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:29.082675 26952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:29.084190 26952 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:29.084728 26952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:29.089406 26952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:29.104287 26952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Bootstrap starting.
I20250815 20:00:29.109402 26952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:29.111465 26952 log.cc:826] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:29.115777 26952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: No bootstrap required, opened a new log
I20250815 20:00:29.132282 26952 raft_consensus.cc:357] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER }
I20250815 20:00:29.132864 26952 raft_consensus.cc:383] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:29.133101 26952 raft_consensus.cc:738] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d563201e36f8477684240c2122f7fa26, State: Initialized, Role: FOLLOWER
I20250815 20:00:29.133860 26952 consensus_queue.cc:260] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER }
I20250815 20:00:29.134368 26952 raft_consensus.cc:397] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 20:00:29.134626 26952 raft_consensus.cc:491] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 20:00:29.134939 26952 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:29.139072 26952 raft_consensus.cc:513] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER }
I20250815 20:00:29.139806 26952 leader_election.cc:304] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d563201e36f8477684240c2122f7fa26; no voters:
I20250815 20:00:29.141572 26952 leader_election.cc:290] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 20:00:29.141829 27002 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 20:00:29.144630 27002 raft_consensus.cc:695] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 LEADER]: Becoming Leader. State: Replica: d563201e36f8477684240c2122f7fa26, State: Running, Role: LEADER
I20250815 20:00:29.145591 27002 consensus_queue.cc:237] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER }
I20250815 20:00:29.153771 27003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d563201e36f8477684240c2122f7fa26. Latest consensus state: current_term: 1 leader_uuid: "d563201e36f8477684240c2122f7fa26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER } }
I20250815 20:00:29.154390 27003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:29.154678 27004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d563201e36f8477684240c2122f7fa26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER } }
I20250815 20:00:29.155171 27004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:29.167184 26952 tablet_replica.cc:331] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: stopping tablet replica
I20250815 20:00:29.167866 26952 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 LEADER]: Raft consensus shutting down.
I20250815 20:00:29.168355 26952 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 20:00:29.170673 26952 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 20:00:29.171187 26952 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 20:00:29.227701 26952 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250815 20:00:30.260037 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 26478
I20250815 20:00:30.307330 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 26611
I20250815 20:00:30.342859 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 26744
I20250815 20:00:30.381191 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:41439
--webserver_interface=127.20.53.126
--webserver_port=40321
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.53.126:41439 with env {}
W20250815 20:00:30.696128 27014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:30.696815 27014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:30.697315 27014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:30.728991 27014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 20:00:30.729305 27014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:30.729600 27014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 20:00:30.729853 27014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 20:00:30.765851 27014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.53.126:41439
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.53.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.53.126
--webserver_port=40321
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:30.767238 27014 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:30.768993 27014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:30.780375 27020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:30.781710 27021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:30.788641 27023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:31.965490 27022 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 20:00:31.965592 27014 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:31.969435 27014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:31.972123 27014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:31.973555 27014 hybrid_clock.cc:648] HybridClock initialized: now 1755288031973508 us; error 62 us; skew 500 ppm
I20250815 20:00:31.974803 27014 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:31.981374 27014 webserver.cc:480] Webserver started at http://127.20.53.126:40321/ using document root <none> and password file <none>
I20250815 20:00:31.982736 27014 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:31.983058 27014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:31.991647 27014 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250815 20:00:31.996585 27030 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:31.997841 27014 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 20:00:31.998203 27014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
uuid: "d563201e36f8477684240c2122f7fa26"
format_stamp: "Formatted at 2025-08-15 20:00:28 on dist-test-slave-d4bs"
I20250815 20:00:32.000213 27014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:32.093416 27014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:32.094964 27014 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:32.095453 27014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:32.167763 27014 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.126:41439
I20250815 20:00:32.167840 27081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.126:41439 every 8 connection(s)
I20250815 20:00:32.170677 27014 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb
I20250815 20:00:32.176134 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 27014
I20250815 20:00:32.178314 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.65:33315
--local_ip_for_outbound_sockets=127.20.53.65
--tserver_master_addrs=127.20.53.126:41439
--webserver_port=40929
--webserver_interface=127.20.53.65
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 20:00:32.184332 27082 sys_catalog.cc:263] Verifying existing consensus state
I20250815 20:00:32.200529 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Bootstrap starting.
I20250815 20:00:32.211464 27082 log.cc:826] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:32.224314 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 20:00:32.225189 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Bootstrap complete.
I20250815 20:00:32.246081 27082 raft_consensus.cc:357] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:32.246789 27082 raft_consensus.cc:738] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d563201e36f8477684240c2122f7fa26, State: Initialized, Role: FOLLOWER
I20250815 20:00:32.247530 27082 consensus_queue.cc:260] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:32.248029 27082 raft_consensus.cc:397] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 20:00:32.248307 27082 raft_consensus.cc:491] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 20:00:32.248629 27082 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 1 FOLLOWER]: Advancing to term 2
I20250815 20:00:32.252835 27082 raft_consensus.cc:513] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:32.253585 27082 leader_election.cc:304] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d563201e36f8477684240c2122f7fa26; no voters:
I20250815 20:00:32.255362 27082 leader_election.cc:290] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 20:00:32.255935 27086 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 20:00:32.258850 27086 raft_consensus.cc:695] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [term 2 LEADER]: Becoming Leader. State: Replica: d563201e36f8477684240c2122f7fa26, State: Running, Role: LEADER
I20250815 20:00:32.259780 27086 consensus_queue.cc:237] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } }
I20250815 20:00:32.260195 27082 sys_catalog.cc:564] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 20:00:32.270670 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d563201e36f8477684240c2122f7fa26. Latest consensus state: current_term: 2 leader_uuid: "d563201e36f8477684240c2122f7fa26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } } }
I20250815 20:00:32.272222 27087 sys_catalog.cc:455] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d563201e36f8477684240c2122f7fa26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d563201e36f8477684240c2122f7fa26" member_type: VOTER last_known_addr { host: "127.20.53.126" port: 41439 } } }
I20250815 20:00:32.272931 27087 sys_catalog.cc:458] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:32.275718 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26 [sys.catalog]: This master's current role is: LEADER
I20250815 20:00:32.280758 27093 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 20:00:32.298134 27093 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=ea4aa1918bdc4d3d87840a8c216578d8]
I20250815 20:00:32.306682 27093 tablet_loader.cc:96] loaded metadata for tablet 9d7b8929d68b4d5d93d86a838c87d857 (table pre_rebuild [id=ea4aa1918bdc4d3d87840a8c216578d8])
I20250815 20:00:32.309805 27093 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 20:00:32.336611 27093 catalog_manager.cc:1349] Generated new cluster ID: ad2e4b95e5404b7e97da149c253869ce
I20250815 20:00:32.336962 27093 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 20:00:32.351292 27105 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250815 20:00:32.360806 27093 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 20:00:32.363276 27093 catalog_manager.cc:1506] Loading token signing keys...
I20250815 20:00:32.391950 27093 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Generated new TSK 0
I20250815 20:00:32.393294 27093 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 20:00:32.541630 27084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:32.542116 27084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:32.542564 27084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:32.575095 27084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:32.575934 27084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.65
I20250815 20:00:32.613543 27084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.65:33315
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.53.65
--webserver_port=40929
--tserver_master_addrs=127.20.53.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.65
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:32.614828 27084 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:32.616434 27084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:32.630813 27111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:32.632047 27112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:34.032912 27110 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 27084
W20250815 20:00:34.146064 27113 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 20:00:34.145196 27084 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.513s user 0.599s sys 0.914s
W20250815 20:00:34.147475 27084 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.515s user 0.599s sys 0.914s
W20250815 20:00:34.147557 27114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:34.147830 27084 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:34.151934 27084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:34.154775 27084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:34.156284 27084 hybrid_clock.cc:648] HybridClock initialized: now 1755288034156221 us; error 87 us; skew 500 ppm
I20250815 20:00:34.157433 27084 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:34.165576 27084 webserver.cc:480] Webserver started at http://127.20.53.65:40929/ using document root <none> and password file <none>
I20250815 20:00:34.167013 27084 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:34.167351 27084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:34.178635 27084 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20250815 20:00:34.185031 27121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:34.186524 27084 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250815 20:00:34.186951 27084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
uuid: "5e7568cc9ace4a7093aa46eada6f8cba"
format_stamp: "Formatted at 2025-08-15 20:00:15 on dist-test-slave-d4bs"
I20250815 20:00:34.190054 27084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:34.268311 27084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:34.269906 27084 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:34.270398 27084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:34.273214 27084 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:34.279759 27128 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 20:00:34.291421 27084 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 20:00:34.291716 27084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20250815 20:00:34.292012 27084 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 20:00:34.297128 27084 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 20:00:34.297358 27084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250815 20:00:34.297896 27128 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Bootstrap starting.
I20250815 20:00:34.497519 27084 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.65:33315
I20250815 20:00:34.497678 27234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.65:33315 every 8 connection(s)
I20250815 20:00:34.501448 27084 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb
I20250815 20:00:34.511426 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 27084
I20250815 20:00:34.513720 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.66:39779
--local_ip_for_outbound_sockets=127.20.53.66
--tserver_master_addrs=127.20.53.126:41439
--webserver_port=38107
--webserver_interface=127.20.53.66
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 20:00:34.568588 27235 heartbeater.cc:344] Connected to a master server at 127.20.53.126:41439
I20250815 20:00:34.569183 27235 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:34.570533 27235 heartbeater.cc:507] Master 127.20.53.126:41439 requested a full tablet report, sending...
I20250815 20:00:34.575454 27047 ts_manager.cc:194] Registered new tserver with Master: 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65:33315)
I20250815 20:00:34.583395 27047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.65:33787
I20250815 20:00:34.681983 27128 log.cc:826] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Log is configured to *not* fsync() on all Append() calls
W20250815 20:00:35.014930 27239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:35.015630 27239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:35.016420 27239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:35.073130 27239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:35.074008 27239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.66
I20250815 20:00:35.109894 27239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.66:39779
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.53.66
--webserver_port=38107
--tserver_master_addrs=127.20.53.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.66
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:35.111305 27239 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:35.113072 27239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:35.126883 27246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:35.588474 27235 heartbeater.cc:499] Master 127.20.53.126:41439 was elected leader, sending a full tablet report...
W20250815 20:00:35.132753 27249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:35.128912 27247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:36.617059 27248 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1485 milliseconds
I20250815 20:00:36.617211 27239 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:36.618744 27239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:36.621692 27239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:36.623250 27239 hybrid_clock.cc:648] HybridClock initialized: now 1755288036623188 us; error 62 us; skew 500 ppm
I20250815 20:00:36.624390 27239 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:36.632364 27239 webserver.cc:480] Webserver started at http://127.20.53.66:38107/ using document root <none> and password file <none>
I20250815 20:00:36.633752 27239 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:36.634075 27239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:36.645427 27239 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20250815 20:00:36.651859 27256 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:36.653281 27239 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20250815 20:00:36.653723 27239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
uuid: "f09629bae44a4fafbfcc26bc20a4d390"
format_stamp: "Formatted at 2025-08-15 20:00:18 on dist-test-slave-d4bs"
I20250815 20:00:36.656648 27239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:36.728230 27239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:36.729861 27239 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:36.730345 27239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:36.733119 27239 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:36.739764 27263 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 20:00:36.751240 27239 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 20:00:36.751546 27239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.002s
I20250815 20:00:36.751839 27239 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 20:00:36.756827 27239 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 20:00:36.757115 27239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250815 20:00:36.757555 27263 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Bootstrap starting.
I20250815 20:00:36.942533 27239 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.66:39779
I20250815 20:00:36.942736 27369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.66:39779 every 8 connection(s)
I20250815 20:00:36.946504 27239 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb
I20250815 20:00:36.953769 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 27239
I20250815 20:00:36.955874 20693 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
/tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.53.67:38931
--local_ip_for_outbound_sockets=127.20.53.67
--tserver_master_addrs=127.20.53.126:41439
--webserver_port=34995
--webserver_interface=127.20.53.67
--builtin_ntp_servers=127.20.53.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 20:00:36.992133 27370 heartbeater.cc:344] Connected to a master server at 127.20.53.126:41439
I20250815 20:00:36.992760 27370 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:36.994167 27370 heartbeater.cc:507] Master 127.20.53.126:41439 requested a full tablet report, sending...
I20250815 20:00:36.998574 27047 ts_manager.cc:194] Registered new tserver with Master: f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779)
I20250815 20:00:37.001627 27047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.66:42711
I20250815 20:00:37.081887 27263 log.cc:826] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Log is configured to *not* fsync() on all Append() calls
W20250815 20:00:37.444581 27374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 20:00:37.445307 27374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 20:00:37.446110 27374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 20:00:37.502833 27374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 20:00:37.504323 27374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.53.67
I20250815 20:00:37.567962 27374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.53.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.53.67:38931
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.53.67
--webserver_port=34995
--tserver_master_addrs=127.20.53.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.53.67
--log_dir=/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 7585
TSAN enabled
I20250815 20:00:37.569914 27374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 20:00:37.572254 27374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 20:00:37.590862 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:37.968639 27128 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 20:00:37.970242 27128 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Bootstrap complete.
I20250815 20:00:37.972779 27128 ts_tablet_manager.cc:1397] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Time spent bootstrapping tablet: real 3.675s user 3.540s sys 0.068s
I20250815 20:00:38.006106 27370 heartbeater.cc:499] Master 127.20.53.126:41439 was elected leader, sending a full tablet report...
I20250815 20:00:38.006801 27128 raft_consensus.cc:357] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:38.023069 27128 raft_consensus.cc:738] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7568cc9ace4a7093aa46eada6f8cba, State: Initialized, Role: FOLLOWER
I20250815 20:00:38.024435 27128 consensus_queue.cc:260] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:38.042665 27128 ts_tablet_manager.cc:1428] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Time spent starting tablet: real 0.069s user 0.024s sys 0.008s
I20250815 20:00:39.269892 27388 raft_consensus.cc:491] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 20:00:39.278265 27388 raft_consensus.cc:513] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:39.288976 27388 leader_election.cc:290] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
W20250815 20:00:39.300696 27123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.53.67:38931: connect: Connection refused (error 111)
W20250815 20:00:39.316047 27123 leader_election.cc:336] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931): Network error: Client connection negotiation failed: client connection to 127.20.53.67:38931: connect: Connection refused (error 111)
I20250815 20:00:39.325270 27325 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390" is_pre_election: true
W20250815 20:00:39.340066 27124 leader_election.cc:343] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779): Illegal state: must be running to vote when last-logged opid is not known
I20250815 20:00:39.340754 27124 leader_election.cc:304] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba; no voters: e52de05298b44607b036b96f02be2fc5, f09629bae44a4fafbfcc26bc20a4d390
I20250815 20:00:39.342500 27388 raft_consensus.cc:2747] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250815 20:00:38.990167 27380 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 27374
W20250815 20:00:37.592690 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 20:00:39.566900 27374 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.977s user 0.611s sys 1.056s
W20250815 20:00:39.568639 27374 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.978s user 0.612s sys 1.056s
W20250815 20:00:39.568840 27383 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1976 milliseconds
W20250815 20:00:39.569416 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 20:00:39.569465 27374 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 20:00:39.574225 27374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 20:00:39.576561 27374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 20:00:39.578033 27374 hybrid_clock.cc:648] HybridClock initialized: now 1755288039577974 us; error 70 us; skew 500 ppm
I20250815 20:00:39.578960 27374 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 20:00:39.586007 27374 webserver.cc:480] Webserver started at http://127.20.53.67:34995/ using document root <none> and password file <none>
I20250815 20:00:39.586997 27374 fs_manager.cc:362] Metadata directory not provided
I20250815 20:00:39.587250 27374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 20:00:39.595837 27374 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250815 20:00:39.601267 27397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 20:00:39.602490 27374 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 20:00:39.602821 27374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
uuid: "e52de05298b44607b036b96f02be2fc5"
format_stamp: "Formatted at 2025-08-15 20:00:20 on dist-test-slave-d4bs"
I20250815 20:00:39.604833 27374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 20:00:39.661824 27374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 20:00:39.663312 27374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 20:00:39.663765 27374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 20:00:39.666921 27374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 20:00:39.673583 27404 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 20:00:39.681532 27374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 20:00:39.681772 27374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.003s
I20250815 20:00:39.682083 27374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 20:00:39.687520 27374 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 20:00:39.687883 27374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.004s sys 0.002s
I20250815 20:00:39.688387 27404 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Bootstrap starting.
I20250815 20:00:39.874243 27374 rpc_server.cc:307] RPC server started. Bound to: 127.20.53.67:38931
I20250815 20:00:39.874413 27510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.53.67:38931 every 8 connection(s)
I20250815 20:00:39.878216 27374 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb
I20250815 20:00:39.879665 20693 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu as pid 27374
I20250815 20:00:39.938612 27511 heartbeater.cc:344] Connected to a master server at 127.20.53.126:41439
I20250815 20:00:39.939155 27511 heartbeater.cc:461] Registering TS with master...
I20250815 20:00:39.940471 27511 heartbeater.cc:507] Master 127.20.53.126:41439 requested a full tablet report, sending...
I20250815 20:00:39.944705 27046 ts_manager.cc:194] Registered new tserver with Master: e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:39.947889 27046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.53.67:44393
I20250815 20:00:39.956897 20693 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 20:00:40.031106 27404 log.cc:826] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Log is configured to *not* fsync() on all Append() calls
I20250815 20:00:40.351923 27263 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 20:00:40.352871 27263 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Bootstrap complete.
I20250815 20:00:40.354239 27263 ts_tablet_manager.cc:1397] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Time spent bootstrapping tablet: real 3.597s user 3.505s sys 0.048s
I20250815 20:00:40.360005 27263 raft_consensus.cc:357] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:40.362201 27263 raft_consensus.cc:738] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09629bae44a4fafbfcc26bc20a4d390, State: Initialized, Role: FOLLOWER
I20250815 20:00:40.362937 27263 consensus_queue.cc:260] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:40.366231 27263 ts_tablet_manager.cc:1428] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20250815 20:00:40.952159 27511 heartbeater.cc:499] Master 127.20.53.126:41439 was elected leader, sending a full tablet report...
I20250815 20:00:41.294978 27526 raft_consensus.cc:491] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 20:00:41.295414 27526 raft_consensus.cc:513] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:41.296947 27526 leader_election.cc:290] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:41.297911 27325 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390" is_pre_election: true
I20250815 20:00:41.298695 27325 raft_consensus.cc:2466] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 1.
I20250815 20:00:41.300375 27124 leader_election.cc:304] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba, f09629bae44a4fafbfcc26bc20a4d390; no voters:
I20250815 20:00:41.302171 27526 raft_consensus.cc:2802] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 20:00:41.302572 27526 raft_consensus.cc:491] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 20:00:41.302971 27526 raft_consensus.cc:3058] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Advancing to term 2
I20250815 20:00:41.310976 27526 raft_consensus.cc:513] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:41.312938 27526 leader_election.cc:290] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 election: Requested vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:41.313800 27325 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390"
I20250815 20:00:41.314357 27325 raft_consensus.cc:3058] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Advancing to term 2
I20250815 20:00:41.323899 27325 raft_consensus.cc:2466] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 2.
I20250815 20:00:41.325225 27124 leader_election.cc:304] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba, f09629bae44a4fafbfcc26bc20a4d390; no voters:
I20250815 20:00:41.326110 27526 raft_consensus.cc:2802] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 FOLLOWER]: Leader election won for term 2
I20250815 20:00:41.315109 27466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e52de05298b44607b036b96f02be2fc5" is_pre_election: true
I20250815 20:00:41.315407 27465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d7b8929d68b4d5d93d86a838c87d857" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e52de05298b44607b036b96f02be2fc5"
W20250815 20:00:41.329099 27123 leader_election.cc:343] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931): Illegal state: must be running to vote when last-logged opid is not known
I20250815 20:00:41.329109 27526 raft_consensus.cc:695] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 LEADER]: Becoming Leader. State: Replica: 5e7568cc9ace4a7093aa46eada6f8cba, State: Running, Role: LEADER
I20250815 20:00:41.330358 27526 consensus_queue.cc:237] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:41.341189 27047 catalog_manager.cc:5582] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba reported cstate change: term changed from 0 to 2, leader changed from <none> to 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65), VOTER 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65) added, VOTER e52de05298b44607b036b96f02be2fc5 (127.20.53.67) added, VOTER f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66) added. New cstate: current_term: 2 leader_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } health_report { overall_health: UNKNOWN } } }
W20250815 20:00:41.715600 27123 consensus_peers.cc:489] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba -> Peer e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931): Couldn't send request to peer e52de05298b44607b036b96f02be2fc5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250815 20:00:41.889281 20693 scanner-internal.cc:458] Time spent opening tablet: real 1.880s user 0.007s sys 0.001s
I20250815 20:00:41.952556 27325 raft_consensus.cc:1273] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 2 FOLLOWER]: Refusing update from remote peer 5e7568cc9ace4a7093aa46eada6f8cba: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250815 20:00:41.954540 27526 consensus_queue.cc:1035] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250815 20:00:42.090449 27190 consensus_queue.cc:237] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:42.096154 27325 raft_consensus.cc:1273] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 2 FOLLOWER]: Refusing update from remote peer 5e7568cc9ace4a7093aa46eada6f8cba: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250815 20:00:42.097930 27541 consensus_queue.cc:1035] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250815 20:00:42.105163 27541 raft_consensus.cc:2953] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER e52de05298b44607b036b96f02be2fc5 (127.20.53.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } }
I20250815 20:00:42.108196 27325 raft_consensus.cc:2953] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER e52de05298b44607b036b96f02be2fc5 (127.20.53.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } }
I20250815 20:00:42.128856 27034 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9d7b8929d68b4d5d93d86a838c87d857 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 20:00:42.133882 27047 catalog_manager.cc:5582] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba reported cstate change: config changed from index -1 to 207, VOTER e52de05298b44607b036b96f02be2fc5 (127.20.53.67) evicted. New cstate: current_term: 2 leader_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } health_report { overall_health: HEALTHY } } }
I20250815 20:00:42.178889 27190 consensus_queue.cc:237] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:42.183418 27541 raft_consensus.cc:2953] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } }
I20250815 20:00:42.190191 27446 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 with delete_type TABLET_DATA_TOMBSTONED (TS e52de05298b44607b036b96f02be2fc5 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:56510
I20250815 20:00:42.198302 27034 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9d7b8929d68b4d5d93d86a838c87d857 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 20:00:42.203138 27046 catalog_manager.cc:5582] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba reported cstate change: config changed from index 207 to 208, VOTER f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66) evicted. New cstate: current_term: 2 leader_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } health_report { overall_health: HEALTHY } } }
W20250815 20:00:42.225212 27032 catalog_manager.cc:4908] TS e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931): delete failed for tablet 9d7b8929d68b4d5d93d86a838c87d857 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d7b8929d68b4d5d93d86a838c87d857 already in progress: opening tablet
I20250815 20:00:42.228910 27404 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 20:00:42.230118 27404 tablet_bootstrap.cc:492] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Bootstrap complete.
I20250815 20:00:42.231981 27404 ts_tablet_manager.cc:1397] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Time spent bootstrapping tablet: real 2.544s user 2.485s sys 0.045s
I20250815 20:00:42.236402 27305 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 with delete_type TABLET_DATA_TOMBSTONED (TS f09629bae44a4fafbfcc26bc20a4d390 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:55726
I20250815 20:00:42.241073 27404 raft_consensus.cc:357] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:42.245366 27404 raft_consensus.cc:738] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52de05298b44607b036b96f02be2fc5, State: Initialized, Role: FOLLOWER
I20250815 20:00:42.246759 27404 consensus_queue.cc:260] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } }
I20250815 20:00:42.252419 27551 tablet_replica.cc:331] stopping tablet replica
I20250815 20:00:42.253767 27551 raft_consensus.cc:2241] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 20:00:42.254577 27551 raft_consensus.cc:2270] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 20:00:42.256600 27404 ts_tablet_manager.cc:1428] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Time spent starting tablet: real 0.024s user 0.015s sys 0.006s
I20250815 20:00:42.263682 27446 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:56510
I20250815 20:00:42.264791 27549 tablet_replica.cc:331] stopping tablet replica
I20250815 20:00:42.265563 27549 raft_consensus.cc:2241] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250815 20:00:42.266098 27549 raft_consensus.cc:2270] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 20:00:42.287124 27551 ts_tablet_manager.cc:1905] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 20:00:42.301170 27549 ts_tablet_manager.cc:1905] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 20:00:42.303092 27551 ts_tablet_manager.cc:1918] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250815 20:00:42.303634 27551 log.cc:1199] T 9d7b8929d68b4d5d93d86a838c87d857 P f09629bae44a4fafbfcc26bc20a4d390: Deleting WAL directory at /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/wal/wals/9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:42.305745 27033 catalog_manager.cc:4928] TS f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779): tablet 9d7b8929d68b4d5d93d86a838c87d857 (table pre_rebuild [id=ea4aa1918bdc4d3d87840a8c216578d8]) successfully deleted
I20250815 20:00:42.315058 27549 ts_tablet_manager.cc:1918] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250815 20:00:42.315402 27549 log.cc:1199] T 9d7b8929d68b4d5d93d86a838c87d857 P e52de05298b44607b036b96f02be2fc5: Deleting WAL directory at /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/wal/wals/9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:42.316820 27032 catalog_manager.cc:4928] TS e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931): tablet 9d7b8929d68b4d5d93d86a838c87d857 (table pre_rebuild [id=ea4aa1918bdc4d3d87840a8c216578d8]) successfully deleted
I20250815 20:00:42.760344 27170 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 20:00:42.762539 27446 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 20:00:42.776999 27305 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d563201e36f8477684240c2122f7fa26 | 127.20.53.126:41439 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.53.84:43335 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5e7568cc9ace4a7093aa46eada6f8cba | 127.20.53.65:33315 | HEALTHY | <none> | 1 | 0
e52de05298b44607b036b96f02be2fc5 | 127.20.53.67:38931 | HEALTHY | <none> | 0 | 0
f09629bae44a4fafbfcc26bc20a4d390 | 127.20.53.66:39779 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.53.65 | experimental | 127.20.53.65:33315
local_ip_for_outbound_sockets | 127.20.53.66 | experimental | 127.20.53.66:39779
local_ip_for_outbound_sockets | 127.20.53.67 | experimental | 127.20.53.67:38931
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.53.65:33315
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.53.66:39779
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.53.67:38931
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.53.84:43335 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 20:00:43.173060 20693 log_verifier.cc:126] Checking tablet 9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:43.463912 20693 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:43.466392 27046 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46952:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 20:00:43.466876 27046 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46952:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 20:00:43.479328 27046 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d563201e36f8477684240c2122f7fa26: Sending DeleteTablet for 1 replicas of tablet 9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:43.481379 20693 test_util.cc:276] Using random seed: -665884411
I20250815 20:00:43.481272 27170 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d7b8929d68b4d5d93d86a838c87d857 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-15 20:00:43 UTC) from {username='slave'} at 127.0.0.1:56422
I20250815 20:00:43.483351 27584 tablet_replica.cc:331] stopping tablet replica
I20250815 20:00:43.484182 27584 raft_consensus.cc:2241] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 LEADER]: Raft consensus shutting down.
I20250815 20:00:43.485019 27584 raft_consensus.cc:2270] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 20:00:43.513522 27046 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47000:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 20:00:43.517019 27046 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 20:00:43.543656 27305 tablet_service.cc:1468] Processing CreateTablet for tablet b1baed73e9f84a9ca309dfd4dc1e4f9e (DEFAULT_TABLE table=post_rebuild [id=f9140c253e4749fbaecef2421a6e9a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:43.545154 27305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1baed73e9f84a9ca309dfd4dc1e4f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:43.545192 27446 tablet_service.cc:1468] Processing CreateTablet for tablet b1baed73e9f84a9ca309dfd4dc1e4f9e (DEFAULT_TABLE table=post_rebuild [id=f9140c253e4749fbaecef2421a6e9a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:43.546505 27446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1baed73e9f84a9ca309dfd4dc1e4f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:43.553602 27584 ts_tablet_manager.cc:1905] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250815 20:00:43.555161 27170 tablet_service.cc:1468] Processing CreateTablet for tablet b1baed73e9f84a9ca309dfd4dc1e4f9e (DEFAULT_TABLE table=post_rebuild [id=f9140c253e4749fbaecef2421a6e9a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 20:00:43.558794 27170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1baed73e9f84a9ca309dfd4dc1e4f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 20:00:43.573377 27584 ts_tablet_manager.cc:1918] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250815 20:00:43.573933 27584 log.cc:1199] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Deleting WAL directory at /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/wal/wals/9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:43.575049 27584 ts_tablet_manager.cc:1939] T 9d7b8929d68b4d5d93d86a838c87d857 P 5e7568cc9ace4a7093aa46eada6f8cba: Deleting consensus metadata
I20250815 20:00:43.578755 27034 catalog_manager.cc:4928] TS 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65:33315): tablet 9d7b8929d68b4d5d93d86a838c87d857 (table pre_rebuild [id=ea4aa1918bdc4d3d87840a8c216578d8]) successfully deleted
I20250815 20:00:43.591907 27591 tablet_bootstrap.cc:492] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390: Bootstrap starting.
I20250815 20:00:43.597965 27592 tablet_bootstrap.cc:492] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba: Bootstrap starting.
I20250815 20:00:43.601504 27593 tablet_bootstrap.cc:492] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5: Bootstrap starting.
I20250815 20:00:43.603519 27591 tablet_bootstrap.cc:654] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:43.604630 27592 tablet_bootstrap.cc:654] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:43.608637 27593 tablet_bootstrap.cc:654] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5: Neither blocks nor log segments found. Creating new log.
I20250815 20:00:43.613286 27592 tablet_bootstrap.cc:492] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba: No bootstrap required, opened a new log
I20250815 20:00:43.613765 27592 ts_tablet_manager.cc:1397] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250815 20:00:43.617086 27592 raft_consensus.cc:357] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.618178 27592 raft_consensus.cc:383] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:43.618568 27592 raft_consensus.cc:738] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7568cc9ace4a7093aa46eada6f8cba, State: Initialized, Role: FOLLOWER
I20250815 20:00:43.620193 27592 consensus_queue.cc:260] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.622485 27591 tablet_bootstrap.cc:492] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390: No bootstrap required, opened a new log
I20250815 20:00:43.622916 27591 ts_tablet_manager.cc:1397] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.008s
I20250815 20:00:43.623605 27593 tablet_bootstrap.cc:492] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5: No bootstrap required, opened a new log
I20250815 20:00:43.624039 27593 ts_tablet_manager.cc:1397] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.006s
I20250815 20:00:43.625723 27591 raft_consensus.cc:357] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.626408 27591 raft_consensus.cc:383] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:43.626703 27591 raft_consensus.cc:738] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09629bae44a4fafbfcc26bc20a4d390, State: Initialized, Role: FOLLOWER
I20250815 20:00:43.627506 27591 consensus_queue.cc:260] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.630395 27593 raft_consensus.cc:357] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.631532 27593 raft_consensus.cc:383] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 20:00:43.631911 27593 raft_consensus.cc:738] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52de05298b44607b036b96f02be2fc5, State: Initialized, Role: FOLLOWER
I20250815 20:00:43.632982 27593 consensus_queue.cc:260] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.636480 27592 ts_tablet_manager.cc:1428] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20250815 20:00:43.636685 27591 ts_tablet_manager.cc:1428] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390: Time spent starting tablet: real 0.013s user 0.003s sys 0.011s
I20250815 20:00:43.641844 27593 ts_tablet_manager.cc:1428] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5: Time spent starting tablet: real 0.018s user 0.004s sys 0.008s
W20250815 20:00:43.657285 27515 tablet.cc:2378] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 20:00:43.677007 27597 raft_consensus.cc:491] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 20:00:43.677429 27597 raft_consensus.cc:513] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.679184 27597 leader_election.cc:290] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:43.680119 27325 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1baed73e9f84a9ca309dfd4dc1e4f9e" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390" is_pre_election: true
I20250815 20:00:43.680379 27466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1baed73e9f84a9ca309dfd4dc1e4f9e" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52de05298b44607b036b96f02be2fc5" is_pre_election: true
I20250815 20:00:43.680784 27325 raft_consensus.cc:2466] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 0.
I20250815 20:00:43.681119 27466 raft_consensus.cc:2466] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 0.
I20250815 20:00:43.682101 27124 leader_election.cc:304] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba, f09629bae44a4fafbfcc26bc20a4d390; no voters:
I20250815 20:00:43.682706 27597 raft_consensus.cc:2802] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 20:00:43.682938 27597 raft_consensus.cc:491] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 20:00:43.683200 27597 raft_consensus.cc:3058] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:43.687856 27597 raft_consensus.cc:513] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.689400 27597 leader_election.cc:290] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 election: Requested vote from peers f09629bae44a4fafbfcc26bc20a4d390 (127.20.53.66:39779), e52de05298b44607b036b96f02be2fc5 (127.20.53.67:38931)
I20250815 20:00:43.690349 27325 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1baed73e9f84a9ca309dfd4dc1e4f9e" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09629bae44a4fafbfcc26bc20a4d390"
I20250815 20:00:43.690924 27325 raft_consensus.cc:3058] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:43.691015 27466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1baed73e9f84a9ca309dfd4dc1e4f9e" candidate_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52de05298b44607b036b96f02be2fc5"
I20250815 20:00:43.691588 27466 raft_consensus.cc:3058] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 0 FOLLOWER]: Advancing to term 1
I20250815 20:00:43.695583 27325 raft_consensus.cc:2466] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 1.
I20250815 20:00:43.696518 27124 leader_election.cc:304] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7568cc9ace4a7093aa46eada6f8cba, f09629bae44a4fafbfcc26bc20a4d390; no voters:
I20250815 20:00:43.697214 27597 raft_consensus.cc:2802] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 FOLLOWER]: Leader election won for term 1
I20250815 20:00:43.698401 27597 raft_consensus.cc:695] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [term 1 LEADER]: Becoming Leader. State: Replica: 5e7568cc9ace4a7093aa46eada6f8cba, State: Running, Role: LEADER
I20250815 20:00:43.698519 27466 raft_consensus.cc:2466] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7568cc9ace4a7093aa46eada6f8cba in term 1.
I20250815 20:00:43.699291 27597 consensus_queue.cc:237] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } }
I20250815 20:00:43.707348 27047 catalog_manager.cc:5582] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7568cc9ace4a7093aa46eada6f8cba (127.20.53.65). New cstate: current_term: 1 leader_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7568cc9ace4a7093aa46eada6f8cba" member_type: VOTER last_known_addr { host: "127.20.53.65" port: 33315 } health_report { overall_health: HEALTHY } } }
W20250815 20:00:43.723282 27371 tablet.cc:2378] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 20:00:43.809130 27236 tablet.cc:2378] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 20:00:43.817248 27466 raft_consensus.cc:1273] T b1baed73e9f84a9ca309dfd4dc1e4f9e P e52de05298b44607b036b96f02be2fc5 [term 1 FOLLOWER]: Refusing update from remote peer 5e7568cc9ace4a7093aa46eada6f8cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 20:00:43.817592 27325 raft_consensus.cc:1273] T b1baed73e9f84a9ca309dfd4dc1e4f9e P f09629bae44a4fafbfcc26bc20a4d390 [term 1 FOLLOWER]: Refusing update from remote peer 5e7568cc9ace4a7093aa46eada6f8cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 20:00:43.819236 27600 consensus_queue.cc:1035] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52de05298b44607b036b96f02be2fc5" member_type: VOTER last_known_addr { host: "127.20.53.67" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 20:00:43.820083 27597 consensus_queue.cc:1035] T b1baed73e9f84a9ca309dfd4dc1e4f9e P 5e7568cc9ace4a7093aa46eada6f8cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09629bae44a4fafbfcc26bc20a4d390" member_type: VOTER last_known_addr { host: "127.20.53.66" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 20:00:43.857863 27608 mvcc.cc:204] Tried to move back new op lower bound from 7189659827459211264 to 7189659826999828480. Current Snapshot: MvccSnapshot[applied={T|T < 7189659827459211264}]
I20250815 20:00:43.864249 27610 mvcc.cc:204] Tried to move back new op lower bound from 7189659827459211264 to 7189659826999828480. Current Snapshot: MvccSnapshot[applied={T|T < 7189659827459211264}]
I20250815 20:00:43.895398 27609 mvcc.cc:204] Tried to move back new op lower bound from 7189659827459211264 to 7189659826999828480. Current Snapshot: MvccSnapshot[applied={T|T < 7189659827459211264}]
W20250815 20:00:44.089658 27507 debug-util.cc:398] Leaking SignalData structure 0x7b08000bada0 after lost signal to thread 27376
W20250815 20:00:44.090759 27507 debug-util.cc:398] Leaking SignalData structure 0x7b08000c87e0 after lost signal to thread 27510
W20250815 20:00:44.147511 27078 debug-util.cc:398] Leaking SignalData structure 0x7b08000aad60 after lost signal to thread 27015
W20250815 20:00:47.427907 27602 meta_cache.cc:1261] Time spent looking up entry by key: real 0.079s user 0.004s sys 0.000s
I20250815 20:00:50.528929 27170 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 20:00:50.538543 27446 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 20:00:50.551249 27305 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d563201e36f8477684240c2122f7fa26 | 127.20.53.126:41439 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.53.84:43335 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5e7568cc9ace4a7093aa46eada6f8cba | 127.20.53.65:33315 | HEALTHY | <none> | 1 | 0
e52de05298b44607b036b96f02be2fc5 | 127.20.53.67:38931 | HEALTHY | <none> | 0 | 0
f09629bae44a4fafbfcc26bc20a4d390 | 127.20.53.66:39779 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.53.65 | experimental | 127.20.53.65:33315
local_ip_for_outbound_sockets | 127.20.53.66 | experimental | 127.20.53.66:39779
local_ip_for_outbound_sockets | 127.20.53.67 | experimental | 127.20.53.67:38931
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.53.65:33315
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.53.66:39779
server_dump_info_path | /tmp/dist-test-taskj_wm5o/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755287859650099-20693-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.53.67:38931
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.53.84:43335 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 20:00:50.781755 20693 log_verifier.cc:126] Checking tablet 9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:50.782181 20693 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 9d7b8929d68b4d5d93d86a838c87d857
I20250815 20:00:50.782330 20693 log_verifier.cc:126] Checking tablet b1baed73e9f84a9ca309dfd4dc1e4f9e
I20250815 20:00:51.588771 20693 log_verifier.cc:177] Verified matching terms for 205 ops in tablet b1baed73e9f84a9ca309dfd4dc1e4f9e
I20250815 20:00:51.621197 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 27084
I20250815 20:00:51.674705 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 27239
I20250815 20:00:51.716881 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 27374
I20250815 20:00:51.757572 20693 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskj_wm5o/build/tsan/bin/kudu with pid 27014
2025-08-15T20:00:51Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (39847 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (39848 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (192096 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST