Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250812 19:57:03.671061 16759 test_util.cc:276] Using random seed: 1907310241
W20250812 19:57:04.883569 16759 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.169s user 0.436s sys 0.730s
W20250812 19:57:04.884025 16759 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.170s user 0.436s sys 0.730s
I20250812 19:57:04.886673 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:57:04.886919 16759 ts_itest-base.cc:116] --------------
I20250812 19:57:04.887115 16759 ts_itest-base.cc:117] 4 tablet servers
I20250812 19:57:04.887327 16759 ts_itest-base.cc:118] 3 replicas per TS
I20250812 19:57:04.887524 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:57:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:57:04Z Disabled control of system clock
I20250812 19:57:04.928732 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44207
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:44207 with env {}
W20250812 19:57:05.248828 16773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:05.249431 16773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:05.249856 16773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:05.280335 16773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:57:05.280675 16773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:05.280922 16773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:57:05.281116 16773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:57:05.316186 16773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:44207
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:05.317519 16773 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:05.319060 16773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:05.330829 16779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:07.148149 16773 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.817s user 0.717s sys 1.099s
W20250812 19:57:05.333093 16780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:06.733152 16778 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 16773
W20250812 19:57:07.148727 16773 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.818s user 0.717s sys 1.100s
W20250812 19:57:07.150748 16782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:07.152945 16781 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1814 milliseconds
I20250812 19:57:07.152978 16773 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:07.154151 16773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:07.156690 16773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:07.158056 16773 hybrid_clock.cc:648] HybridClock initialized: now 1755028627158001 us; error 62 us; skew 500 ppm
I20250812 19:57:07.158825 16773 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:07.165338 16773 webserver.cc:480] Webserver started at http://127.16.93.254:44407/ using document root <none> and password file <none>
I20250812 19:57:07.166275 16773 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:07.166492 16773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:07.166919 16773 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:07.171201 16773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7a16885e27624691a6c267899a467530"
format_stamp: "Formatted at 2025-08-12 19:57:07 on dist-test-slave-wz96"
I20250812 19:57:07.172325 16773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7a16885e27624691a6c267899a467530"
format_stamp: "Formatted at 2025-08-12 19:57:07 on dist-test-slave-wz96"
I20250812 19:57:07.179867 16773 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250812 19:57:07.185930 16789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:07.187005 16773 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250812 19:57:07.187337 16773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7a16885e27624691a6c267899a467530"
format_stamp: "Formatted at 2025-08-12 19:57:07 on dist-test-slave-wz96"
I20250812 19:57:07.187677 16773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:07.252418 16773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:07.253902 16773 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:07.254354 16773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:07.330730 16773 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:44207
I20250812 19:57:07.330791 16840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:44207 every 8 connection(s)
I20250812 19:57:07.333400 16773 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:57:07.337399 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 16773
I20250812 19:57:07.337796 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:57:07.338919 16841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:07.359540 16841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Bootstrap starting.
I20250812 19:57:07.365778 16841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:07.367462 16841 log.cc:826] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:07.372756 16841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: No bootstrap required, opened a new log
I20250812 19:57:07.391866 16841 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:07.392519 16841 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:07.392809 16841 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a16885e27624691a6c267899a467530, State: Initialized, Role: FOLLOWER
I20250812 19:57:07.393425 16841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:07.393898 16841 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:57:07.394153 16841 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:57:07.394459 16841 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:07.399004 16841 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:07.399701 16841 leader_election.cc:304] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a16885e27624691a6c267899a467530; no voters:
I20250812 19:57:07.402526 16841 leader_election.cc:290] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:57:07.403278 16846 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:07.405825 16846 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 LEADER]: Becoming Leader. State: Replica: 7a16885e27624691a6c267899a467530, State: Running, Role: LEADER
I20250812 19:57:07.406663 16846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:07.408679 16841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:57:07.418241 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a16885e27624691a6c267899a467530. Latest consensus state: current_term: 1 leader_uuid: "7a16885e27624691a6c267899a467530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } } }
I20250812 19:57:07.418743 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:07.420274 16847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a16885e27624691a6c267899a467530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } } }
I20250812 19:57:07.421196 16847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:07.424415 16853 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:57:07.436945 16853 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:57:07.463510 16853 catalog_manager.cc:1349] Generated new cluster ID: 4a68c17df7f941ac84585f418e2f0960
I20250812 19:57:07.463836 16853 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:57:07.532019 16853 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:57:07.534086 16853 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:57:07.551318 16853 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Generated new TSK 0
I20250812 19:57:07.552376 16853 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:57:07.578881 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--builtin_ntp_servers=127.16.93.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 19:57:07.907104 16865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:07.907622 16865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:07.908111 16865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:07.939425 16865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:07.940310 16865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:57:07.976048 16865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:07.977754 16865 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:07.979740 16865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:07.995440 16871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:09.398022 16870 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 16865
W20250812 19:57:09.776024 16870 kernel_stack_watchdog.cc:198] Thread 16865 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:57:07.996974 16872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:09.777632 16865 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.781s user 0.589s sys 1.132s
W20250812 19:57:09.778098 16865 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.782s user 0.589s sys 1.133s
W20250812 19:57:09.778331 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:09.781829 16873 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1781 milliseconds
I20250812 19:57:09.781909 16865 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:09.783331 16865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:09.785895 16865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:09.787379 16865 hybrid_clock.cc:648] HybridClock initialized: now 1755028629787326 us; error 47 us; skew 500 ppm
I20250812 19:57:09.788421 16865 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:09.796344 16865 webserver.cc:480] Webserver started at http://127.16.93.193:35453/ using document root <none> and password file <none>
I20250812 19:57:09.797626 16865 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:09.797905 16865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:09.798488 16865 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:09.805960 16865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cadfbad94c55478b9c448546d31813f8"
format_stamp: "Formatted at 2025-08-12 19:57:09 on dist-test-slave-wz96"
I20250812 19:57:09.807475 16865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cadfbad94c55478b9c448546d31813f8"
format_stamp: "Formatted at 2025-08-12 19:57:09 on dist-test-slave-wz96"
I20250812 19:57:09.817122 16865 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.002s
I20250812 19:57:09.824821 16881 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:09.825889 16865 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250812 19:57:09.826195 16865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cadfbad94c55478b9c448546d31813f8"
format_stamp: "Formatted at 2025-08-12 19:57:09 on dist-test-slave-wz96"
I20250812 19:57:09.826514 16865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:09.883208 16865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:09.884666 16865 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:09.885083 16865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:09.887650 16865 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:09.892710 16865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:09.892899 16865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:09.893172 16865 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:09.893332 16865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:10.081291 16865 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:40943
I20250812 19:57:10.081881 16993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:40943 every 8 connection(s)
I20250812 19:57:10.084154 16865 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:57:10.086850 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 16865
I20250812 19:57:10.087304 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:57:10.097044 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--builtin_ntp_servers=127.16.93.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:57:10.121363 16994 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:10.121892 16994 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:10.123214 16994 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:10.126854 16806 ts_manager.cc:194] Registered new tserver with Master: cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943)
I20250812 19:57:10.129642 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:38925
W20250812 19:57:10.417727 16998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:10.418256 16998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:10.418781 16998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:10.449585 16998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:10.450431 16998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:57:10.484215 16998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:10.485517 16998 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:10.487018 16998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:10.498924 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:11.133869 16994 heartbeater.cc:499] Master 127.16.93.254:44207 was elected leader, sending a full tablet report...
W20250812 19:57:10.500015 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:11.711040 17006 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1206 milliseconds
W20250812 19:57:11.712071 16998 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.213s user 0.375s sys 0.832s
W20250812 19:57:11.712445 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:11.712524 16998 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.214s user 0.375s sys 0.832s
I20250812 19:57:11.712930 16998 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:11.714469 16998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:11.717211 16998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:11.718770 16998 hybrid_clock.cc:648] HybridClock initialized: now 1755028631718703 us; error 36 us; skew 500 ppm
I20250812 19:57:11.719940 16998 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:11.728971 16998 webserver.cc:480] Webserver started at http://127.16.93.194:44377/ using document root <none> and password file <none>
I20250812 19:57:11.730284 16998 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:11.730587 16998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:11.731256 16998 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:11.738714 16998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "157206c188a14e3ba68afa99b2f7f127"
format_stamp: "Formatted at 2025-08-12 19:57:11 on dist-test-slave-wz96"
I20250812 19:57:11.740267 16998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "157206c188a14e3ba68afa99b2f7f127"
format_stamp: "Formatted at 2025-08-12 19:57:11 on dist-test-slave-wz96"
I20250812 19:57:11.750598 16998 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20250812 19:57:11.758893 17014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:11.760123 16998 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20250812 19:57:11.760528 16998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "157206c188a14e3ba68afa99b2f7f127"
format_stamp: "Formatted at 2025-08-12 19:57:11 on dist-test-slave-wz96"
I20250812 19:57:11.760995 16998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:11.841645 16998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:11.842991 16998 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:11.843361 16998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:11.846094 16998 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:11.850616 16998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:11.850821 16998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:11.851022 16998 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:11.851150 16998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:11.998296 16998 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:35867
I20250812 19:57:11.998400 17126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:35867 every 8 connection(s)
I20250812 19:57:12.000823 16998 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:57:12.008478 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 16998
I20250812 19:57:12.008899 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 19:57:12.014755 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--builtin_ntp_servers=127.16.93.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:57:12.023730 17127 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:12.024186 17127 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:12.025239 17127 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:12.027326 16806 ts_manager.cc:194] Registered new tserver with Master: 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:12.028685 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:52009
W20250812 19:57:12.333221 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:12.333864 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:12.334610 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:12.376601 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:12.377375 17131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:57:12.412134 17131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:12.413457 17131 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:12.414923 17131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:12.426541 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:13.031955 17127 heartbeater.cc:499] Master 127.16.93.254:44207 was elected leader, sending a full tablet report...
W20250812 19:57:12.427585 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:13.639833 17140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:13.641893 17139 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1209 milliseconds
I20250812 19:57:13.641994 17131 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:13.643179 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:13.645269 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:13.646610 17131 hybrid_clock.cc:648] HybridClock initialized: now 1755028633646569 us; error 48 us; skew 500 ppm
I20250812 19:57:13.647375 17131 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:13.653595 17131 webserver.cc:480] Webserver started at http://127.16.93.195:43833/ using document root <none> and password file <none>
I20250812 19:57:13.654464 17131 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:13.654656 17131 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:13.655109 17131 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:13.659260 17131 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7fb6a25b01a64037923c34f77051d99a"
format_stamp: "Formatted at 2025-08-12 19:57:13 on dist-test-slave-wz96"
I20250812 19:57:13.660367 17131 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7fb6a25b01a64037923c34f77051d99a"
format_stamp: "Formatted at 2025-08-12 19:57:13 on dist-test-slave-wz96"
I20250812 19:57:13.667448 17131 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250812 19:57:13.672748 17147 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:13.673693 17131 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 19:57:13.674015 17131 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7fb6a25b01a64037923c34f77051d99a"
format_stamp: "Formatted at 2025-08-12 19:57:13 on dist-test-slave-wz96"
I20250812 19:57:13.674471 17131 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:13.728495 17131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:13.729902 17131 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:13.730350 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:13.732885 17131 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:13.736747 17131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:13.736930 17131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:13.737301 17131 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:13.737452 17131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:13.879734 17131 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:39893
I20250812 19:57:13.879845 17260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:39893 every 8 connection(s)
I20250812 19:57:13.882335 17131 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:57:13.891463 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17131
I20250812 19:57:13.892159 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 19:57:13.899205 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.196:0
--local_ip_for_outbound_sockets=127.16.93.196
--webserver_interface=127.16.93.196
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--builtin_ntp_servers=127.16.93.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:57:13.905711 17261 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:13.906123 17261 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:13.907150 17261 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:13.909278 16806 ts_manager.cc:194] Registered new tserver with Master: 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893)
I20250812 19:57:13.911136 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:39349
W20250812 19:57:14.212699 17265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:14.213151 17265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:14.213665 17265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:14.246064 17265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:14.246872 17265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.196
I20250812 19:57:14.281349 17265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.93.196
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.196
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:14.282740 17265 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:14.284265 17265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:14.296655 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:14.914459 17261 heartbeater.cc:499] Master 127.16.93.254:44207 was elected leader, sending a full tablet report...
W20250812 19:57:14.298832 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:15.581694 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:15.583669 17273 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1281 milliseconds
I20250812 19:57:15.583796 17265 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:15.584985 17265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:15.587628 17265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:15.589051 17265 hybrid_clock.cc:648] HybridClock initialized: now 1755028635588999 us; error 60 us; skew 500 ppm
I20250812 19:57:15.589910 17265 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:15.596557 17265 webserver.cc:480] Webserver started at http://127.16.93.196:46779/ using document root <none> and password file <none>
I20250812 19:57:15.597523 17265 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:15.597723 17265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:15.598191 17265 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:15.603008 17265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f08f0f98d6b64950a82bf97f0935e4ea"
format_stamp: "Formatted at 2025-08-12 19:57:15 on dist-test-slave-wz96"
I20250812 19:57:15.604624 17265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f08f0f98d6b64950a82bf97f0935e4ea"
format_stamp: "Formatted at 2025-08-12 19:57:15 on dist-test-slave-wz96"
I20250812 19:57:15.611690 17265 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20250812 19:57:15.617738 17282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:15.618746 17265 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250812 19:57:15.619037 17265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f08f0f98d6b64950a82bf97f0935e4ea"
format_stamp: "Formatted at 2025-08-12 19:57:15 on dist-test-slave-wz96"
I20250812 19:57:15.619321 17265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:15.694180 17265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:15.695621 17265 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:15.696025 17265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:15.698426 17265 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:15.702380 17265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:15.702579 17265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:15.702785 17265 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:15.702917 17265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:15.858741 17265 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.196:45543
I20250812 19:57:15.858824 17394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.196:45543 every 8 connection(s)
I20250812 19:57:15.861454 17265 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250812 19:57:15.863112 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17265
I20250812 19:57:15.863709 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250812 19:57:15.890244 17395 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:15.890777 17395 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:15.892048 17395 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:15.894860 16806 ts_manager.cc:194] Registered new tserver with Master: f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196:45543)
I20250812 19:57:15.896323 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.196:47405
I20250812 19:57:15.900844 16759 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250812 19:57:15.947249 16806 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35700:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250812 19:57:16.057502 17196 tablet_service.cc:1468] Processing CreateTablet for tablet 252544c7797f4f028ead00e021b1f280 (DEFAULT_TABLE table=TestTable [id=3d0dc9cf72e04190ab2847db129ebd90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:16.057566 16929 tablet_service.cc:1468] Processing CreateTablet for tablet 252544c7797f4f028ead00e021b1f280 (DEFAULT_TABLE table=TestTable [id=3d0dc9cf72e04190ab2847db129ebd90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:16.059494 17196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252544c7797f4f028ead00e021b1f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:16.059490 16929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252544c7797f4f028ead00e021b1f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:16.061519 17062 tablet_service.cc:1468] Processing CreateTablet for tablet 252544c7797f4f028ead00e021b1f280 (DEFAULT_TABLE table=TestTable [id=3d0dc9cf72e04190ab2847db129ebd90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:16.063156 17062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252544c7797f4f028ead00e021b1f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:16.094569 17414 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Bootstrap starting.
I20250812 19:57:16.097841 17415 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: Bootstrap starting.
I20250812 19:57:16.099604 17416 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: Bootstrap starting.
I20250812 19:57:16.108192 17416 tablet_bootstrap.cc:654] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:16.111042 17416 log.cc:826] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:16.112771 17414 tablet_bootstrap.cc:654] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:16.113447 17415 tablet_bootstrap.cc:654] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:16.115062 17414 log.cc:826] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:16.119675 17415 log.cc:826] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:16.123670 17416 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: No bootstrap required, opened a new log
I20250812 19:57:16.124082 17416 ts_tablet_manager.cc:1397] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.010s
I20250812 19:57:16.124429 17414 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: No bootstrap required, opened a new log
I20250812 19:57:16.124859 17414 ts_tablet_manager.cc:1397] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.004s
I20250812 19:57:16.126542 17415 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: No bootstrap required, opened a new log
I20250812 19:57:16.127007 17415 ts_tablet_manager.cc:1397] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.007s
I20250812 19:57:16.144310 17414 raft_consensus.cc:357] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.145051 17414 raft_consensus.cc:383] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:16.145320 17414 raft_consensus.cc:738] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb6a25b01a64037923c34f77051d99a, State: Initialized, Role: FOLLOWER
I20250812 19:57:16.146095 17414 consensus_queue.cc:260] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.150876 17414 ts_tablet_manager.cc:1428] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Time spent starting tablet: real 0.026s user 0.016s sys 0.008s
I20250812 19:57:16.151866 17416 raft_consensus.cc:357] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.152791 17416 raft_consensus.cc:383] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:16.153105 17416 raft_consensus.cc:738] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadfbad94c55478b9c448546d31813f8, State: Initialized, Role: FOLLOWER
I20250812 19:57:16.154923 17415 raft_consensus.cc:357] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.155063 17416 consensus_queue.cc:260] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.155802 17415 raft_consensus.cc:383] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:16.156091 17415 raft_consensus.cc:738] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157206c188a14e3ba68afa99b2f7f127, State: Initialized, Role: FOLLOWER
I20250812 19:57:16.158656 17415 consensus_queue.cc:260] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.159622 17416 ts_tablet_manager.cc:1428] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: Time spent starting tablet: real 0.035s user 0.027s sys 0.009s
I20250812 19:57:16.163913 17415 ts_tablet_manager.cc:1428] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: Time spent starting tablet: real 0.037s user 0.036s sys 0.000s
I20250812 19:57:16.244467 17421 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:57:16.244959 17421 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.247386 17421 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893), 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:16.261567 17216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "cadfbad94c55478b9c448546d31813f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6a25b01a64037923c34f77051d99a" is_pre_election: true
I20250812 19:57:16.262032 17082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "cadfbad94c55478b9c448546d31813f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157206c188a14e3ba68afa99b2f7f127" is_pre_election: true
W20250812 19:57:16.262632 17128 tablet.cc:2378] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:16.262553 17216 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cadfbad94c55478b9c448546d31813f8 in term 0.
I20250812 19:57:16.263233 17082 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cadfbad94c55478b9c448546d31813f8 in term 0.
I20250812 19:57:16.264377 16885 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb6a25b01a64037923c34f77051d99a, cadfbad94c55478b9c448546d31813f8; no voters:
I20250812 19:57:16.265090 17421 raft_consensus.cc:2802] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:57:16.265434 17421 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:57:16.265679 17421 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:16.318768 17421 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.320360 17421 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893), 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:16.321120 17216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "cadfbad94c55478b9c448546d31813f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6a25b01a64037923c34f77051d99a"
I20250812 19:57:16.321600 17216 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:16.321640 17082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "cadfbad94c55478b9c448546d31813f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157206c188a14e3ba68afa99b2f7f127"
I20250812 19:57:16.322188 17082 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 0 FOLLOWER]: Advancing to term 1
W20250812 19:57:16.363016 16995 tablet.cc:2378] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 19:57:16.390028 17262 tablet.cc:2378] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:16.423961 17082 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cadfbad94c55478b9c448546d31813f8 in term 1.
I20250812 19:57:16.423954 17216 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cadfbad94c55478b9c448546d31813f8 in term 1.
I20250812 19:57:16.425127 16885 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb6a25b01a64037923c34f77051d99a, cadfbad94c55478b9c448546d31813f8; no voters:
I20250812 19:57:16.425751 17421 raft_consensus.cc:2802] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:16.427863 17421 raft_consensus.cc:695] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [term 1 LEADER]: Becoming Leader. State: Replica: cadfbad94c55478b9c448546d31813f8, State: Running, Role: LEADER
I20250812 19:57:16.428776 17421 consensus_queue.cc:237] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:16.441728 16804 catalog_manager.cc:5582] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cadfbad94c55478b9c448546d31813f8 (127.16.93.193). New cstate: current_term: 1 leader_uuid: "cadfbad94c55478b9c448546d31813f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } health_report { overall_health: UNKNOWN } } }
I20250812 19:57:16.528648 16759 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250812 19:57:16.532259 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cadfbad94c55478b9c448546d31813f8 to finish bootstrapping
I20250812 19:57:16.544910 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 157206c188a14e3ba68afa99b2f7f127 to finish bootstrapping
I20250812 19:57:16.555357 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7fb6a25b01a64037923c34f77051d99a to finish bootstrapping
I20250812 19:57:16.566866 16759 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250812 19:57:16.569993 16759 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "252544c7797f4f028ead00e021b1f280"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "cadfbad94c55478b9c448546d31813f8"
rpc_addresses {
host: "127.16.93.193"
port: 40943
}
}
ts_infos {
permanent_uuid: "7fb6a25b01a64037923c34f77051d99a"
rpc_addresses {
host: "127.16.93.195"
port: 39893
}
}
ts_infos {
permanent_uuid: "157206c188a14e3ba68afa99b2f7f127"
rpc_addresses {
host: "127.16.93.194"
port: 35867
}
}
I20250812 19:57:16.900144 17395 heartbeater.cc:499] Master 127.16.93.254:44207 was elected leader, sending a full tablet report...
I20250812 19:57:16.963035 17429 consensus_queue.cc:1035] T 252544c7797f4f028ead00e021b1f280 P cadfbad94c55478b9c448546d31813f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 19:57:16.977061 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 16865
W20250812 19:57:17.002017 17150 connection.cc:537] server connection from 127.16.93.193:54617 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250812 19:57:17.002002 17017 connection.cc:537] server connection from 127.16.93.193:38705 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250812 19:57:17.002612 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 16773
W20250812 19:57:17.005115 17123 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6bc0 after lost signal to thread 16999
I20250812 19:57:17.042104 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44207
--webserver_interface=127.16.93.254
--webserver_port=44407
--builtin_ntp_servers=127.16.93.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:44207 with env {}
W20250812 19:57:17.181273 17127 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:44207 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:44207: connect: Connection refused (error 111)
W20250812 19:57:17.376889 17435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:17.377627 17435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:17.378198 17435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:17.411618 17435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:57:17.411936 17435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:17.412190 17435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:57:17.412420 17435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:57:17.458019 17435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:44207
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=44407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:17.459324 17435 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:17.461228 17435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:17.473064 17442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:17.914650 17395 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:44207 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:44207: connect: Connection refused (error 111)
W20250812 19:57:18.003103 17261 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:44207 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:44207: connect: Connection refused (error 111)
I20250812 19:57:18.085485 17450 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:57:18.086203 17450 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:18.116057 17450 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943), 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893)
W20250812 19:57:18.131647 17016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111)
I20250812 19:57:18.142876 17216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "157206c188a14e3ba68afa99b2f7f127" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6a25b01a64037923c34f77051d99a" is_pre_election: true
W20250812 19:57:18.151131 17016 leader_election.cc:336] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943): Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111)
I20250812 19:57:18.152060 17016 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 157206c188a14e3ba68afa99b2f7f127; no voters: 7fb6a25b01a64037923c34f77051d99a, cadfbad94c55478b9c448546d31813f8
I20250812 19:57:18.153091 17450 raft_consensus.cc:2747] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250812 19:57:18.493422 17455 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cadfbad94c55478b9c448546d31813f8)
I20250812 19:57:18.494199 17455 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
W20250812 19:57:18.499550 17150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111)
I20250812 19:57:18.520183 17455 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943), 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
W20250812 19:57:18.532338 17150 leader_election.cc:336] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943): Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111)
I20250812 19:57:18.541786 17082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "7fb6a25b01a64037923c34f77051d99a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "157206c188a14e3ba68afa99b2f7f127" is_pre_election: true
I20250812 19:57:18.553344 17082 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb6a25b01a64037923c34f77051d99a in term 1.
I20250812 19:57:18.555037 17151 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 157206c188a14e3ba68afa99b2f7f127, 7fb6a25b01a64037923c34f77051d99a; no voters: cadfbad94c55478b9c448546d31813f8
I20250812 19:57:18.556444 17455 raft_consensus.cc:2802] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250812 19:57:18.556809 17455 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 1 FOLLOWER]: Starting leader election (detected failure of leader cadfbad94c55478b9c448546d31813f8)
I20250812 19:57:18.557157 17455 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:57:18.567696 17455 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:18.573745 17082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "7fb6a25b01a64037923c34f77051d99a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "157206c188a14e3ba68afa99b2f7f127"
I20250812 19:57:18.574322 17082 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 1 FOLLOWER]: Advancing to term 2
W20250812 19:57:18.582161 17150 leader_election.cc:336] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943): Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111)
I20250812 19:57:18.583420 17082 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb6a25b01a64037923c34f77051d99a in term 2.
I20250812 19:57:18.584547 17151 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 157206c188a14e3ba68afa99b2f7f127, 7fb6a25b01a64037923c34f77051d99a; no voters: cadfbad94c55478b9c448546d31813f8
I20250812 19:57:18.587170 17455 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 2 election: Requested vote from peers cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943), 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:18.587577 17459 raft_consensus.cc:2802] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 FOLLOWER]: Leader election won for term 2
I20250812 19:57:18.588493 17459 raft_consensus.cc:695] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 LEADER]: Becoming Leader. State: Replica: 7fb6a25b01a64037923c34f77051d99a, State: Running, Role: LEADER
I20250812 19:57:18.594033 17459 consensus_queue.cc:237] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:19.012527 17082 raft_consensus.cc:1273] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Refusing update from remote peer 7fb6a25b01a64037923c34f77051d99a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250812 19:57:19.014963 17459 consensus_queue.cc:1035] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Connected to new peer: Peer: permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20250812 19:57:19.021828 17150 consensus_peers.cc:489] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a -> Peer cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943): Couldn't send request to peer cadfbad94c55478b9c448546d31813f8. Status: Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250812 19:57:17.479036 17443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:19.138139 17444 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1663 milliseconds
W20250812 19:57:19.138778 17445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:19.138782 17435 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:19.142525 17435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:19.145530 17435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:19.147046 17435 hybrid_clock.cc:648] HybridClock initialized: now 1755028639146986 us; error 62 us; skew 500 ppm
I20250812 19:57:19.147866 17435 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:19.153894 17435 webserver.cc:480] Webserver started at http://127.16.93.254:44407/ using document root <none> and password file <none>
I20250812 19:57:19.154917 17435 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:19.155162 17435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:19.162750 17435 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250812 19:57:19.167155 17471 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:19.168149 17435 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250812 19:57:19.168452 17435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7a16885e27624691a6c267899a467530"
format_stamp: "Formatted at 2025-08-12 19:57:07 on dist-test-slave-wz96"
I20250812 19:57:19.170356 17435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:19.224824 17435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:19.226293 17435 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:19.226738 17435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:19.296878 17435 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:44207
I20250812 19:57:19.296990 17522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:44207 every 8 connection(s)
I20250812 19:57:19.299775 17435 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:57:19.306208 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17435
I20250812 19:57:19.306831 16759 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 7fb6a25b01a64037923c34f77051d99a
I20250812 19:57:19.312525 17523 sys_catalog.cc:263] Verifying existing consensus state
I20250812 19:57:19.318876 17523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Bootstrap starting.
I20250812 19:57:19.370555 17523 log.cc:826] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:19.400813 17523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:57:19.401850 17523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Bootstrap complete.
I20250812 19:57:19.431391 17523 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:19.433917 17523 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a16885e27624691a6c267899a467530, State: Initialized, Role: FOLLOWER
I20250812 19:57:19.434892 17523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:19.435597 17523 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:57:19.435954 17523 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:57:19.436271 17523 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:57:19.445221 17523 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:19.446218 17523 leader_election.cc:304] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a16885e27624691a6c267899a467530; no voters:
I20250812 19:57:19.448827 17523 leader_election.cc:290] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 19:57:19.449105 17527 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 19:57:19.452405 17527 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [term 2 LEADER]: Becoming Leader. State: Replica: 7a16885e27624691a6c267899a467530, State: Running, Role: LEADER
I20250812 19:57:19.453331 17527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } }
I20250812 19:57:19.454182 17523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:57:19.463989 17528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7a16885e27624691a6c267899a467530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } } }
I20250812 19:57:19.464900 17528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:19.466311 17529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a16885e27624691a6c267899a467530. Latest consensus state: current_term: 2 leader_uuid: "7a16885e27624691a6c267899a467530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a16885e27624691a6c267899a467530" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44207 } } }
I20250812 19:57:19.466961 17529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530 [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:19.479689 17534 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:57:19.493212 17534 catalog_manager.cc:671] Loaded metadata for table TestTable [id=3d0dc9cf72e04190ab2847db129ebd90]
I20250812 19:57:19.502334 17534 tablet_loader.cc:96] loaded metadata for tablet 252544c7797f4f028ead00e021b1f280 (table TestTable [id=3d0dc9cf72e04190ab2847db129ebd90])
I20250812 19:57:19.504530 17534 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:57:19.510757 17534 catalog_manager.cc:1261] Loaded cluster ID: 4a68c17df7f941ac84585f418e2f0960
I20250812 19:57:19.511236 17534 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:57:19.521220 17534 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:57:19.525560 17544 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250812 19:57:19.527047 17534 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7a16885e27624691a6c267899a467530: Loaded TSK: 0
I20250812 19:57:19.528246 17534 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 19:57:19.694098 17525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:19.694687 17525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:19.726521 17525 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250812 19:57:19.979377 17395 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:19.986815 17488 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" instance_seqno: 1755028635820335) as {username='slave'} at 127.16.93.196:48173; Asking this server to re-register.
I20250812 19:57:19.989558 17395 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:19.990362 17395 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:19.993201 17487 ts_manager.cc:194] Registered new tserver with Master: f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196:45543)
I20250812 19:57:20.123936 17127 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:20.128281 17487 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" instance_seqno: 1755028631964383) as {username='slave'} at 127.16.93.194:48609; Asking this server to re-register.
I20250812 19:57:20.130419 17127 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:20.131242 17127 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:20.135155 17487 ts_manager.cc:194] Registered new tserver with Master: 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:20.141942 17487 catalog_manager.cc:5582] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 reported cstate change: term changed from 1 to 2, leader changed from cadfbad94c55478b9c448546d31813f8 (127.16.93.193) to 7fb6a25b01a64037923c34f77051d99a (127.16.93.195). New cstate: current_term: 2 leader_uuid: "7fb6a25b01a64037923c34f77051d99a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } }
I20250812 19:57:20.172950 17261 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44207
I20250812 19:57:20.178608 17488 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" instance_seqno: 1755028633846598) as {username='slave'} at 127.16.93.195:42667; Asking this server to re-register.
I20250812 19:57:20.180428 17261 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:20.181427 17261 heartbeater.cc:507] Master 127.16.93.254:44207 requested a full tablet report, sending...
I20250812 19:57:20.186393 17488 ts_manager.cc:194] Registered new tserver with Master: 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893)
W20250812 19:57:21.122249 17525 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.353s user 0.492s sys 0.797s
W20250812 19:57:21.122619 17525 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.353s user 0.492s sys 0.797s
I20250812 19:57:21.190228 17216 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "7fb6a25b01a64037923c34f77051d99a"
tablet_id: "252544c7797f4f028ead00e021b1f280"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "157206c188a14e3ba68afa99b2f7f127"
}
peers {
permanent_uuid: "7fb6a25b01a64037923c34f77051d99a"
}
}
from {username='slave'} at 127.0.0.1:45972
W20250812 19:57:21.191556 17216 raft_consensus.cc:2216] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true
I20250812 19:57:21.192736 17216 raft_consensus.cc:3053] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 LEADER]: Stepping down as leader of term 2
I20250812 19:57:21.193063 17216 raft_consensus.cc:738] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7fb6a25b01a64037923c34f77051d99a, State: Running, Role: LEADER
I20250812 19:57:21.193787 17216 consensus_queue.cc:260] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:21.194991 17216 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:57:22.589939 17575 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7fb6a25b01a64037923c34f77051d99a)
I20250812 19:57:22.590507 17575 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } }
I20250812 19:57:22.592235 17575 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943), 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893)
I20250812 19:57:22.594146 17216 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "157206c188a14e3ba68afa99b2f7f127" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7fb6a25b01a64037923c34f77051d99a" is_pre_election: true
W20250812 19:57:22.597433 17016 leader_election.cc:336] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cadfbad94c55478b9c448546d31813f8 (127.16.93.193:40943): Network error: Client connection negotiation failed: client connection to 127.16.93.193:40943: connect: Connection refused (error 111)
I20250812 19:57:22.597775 17016 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 157206c188a14e3ba68afa99b2f7f127; no voters: 7fb6a25b01a64037923c34f77051d99a, cadfbad94c55478b9c448546d31813f8
I20250812 19:57:22.598343 17575 raft_consensus.cc:2747] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250812 19:57:22.703207 17578 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250812 19:57:22.703779 17578 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true
I20250812 19:57:22.705291 17578 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:22.706338 17082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "7fb6a25b01a64037923c34f77051d99a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "157206c188a14e3ba68afa99b2f7f127" is_pre_election: true
I20250812 19:57:22.706898 17082 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb6a25b01a64037923c34f77051d99a in term 2.
I20250812 19:57:22.707996 17151 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 157206c188a14e3ba68afa99b2f7f127, 7fb6a25b01a64037923c34f77051d99a; no voters:
I20250812 19:57:22.708561 17578 raft_consensus.cc:2802] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250812 19:57:22.708830 17578 raft_consensus.cc:491] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250812 19:57:22.709090 17578 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 3 FOLLOWER]: Advancing to term 4
I20250812 19:57:22.721112 17578 raft_consensus.cc:513] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true
I20250812 19:57:22.722545 17578 leader_election.cc:290] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 4 election: Requested vote from peers 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194:35867)
I20250812 19:57:22.723455 17082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "252544c7797f4f028ead00e021b1f280" candidate_uuid: "7fb6a25b01a64037923c34f77051d99a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "157206c188a14e3ba68afa99b2f7f127"
I20250812 19:57:22.724279 17082 raft_consensus.cc:3058] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 2 FOLLOWER]: Advancing to term 4
I20250812 19:57:22.739064 17082 raft_consensus.cc:2466] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb6a25b01a64037923c34f77051d99a in term 4.
I20250812 19:57:22.740026 17151 leader_election.cc:304] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 157206c188a14e3ba68afa99b2f7f127, 7fb6a25b01a64037923c34f77051d99a; no voters:
I20250812 19:57:22.740676 17578 raft_consensus.cc:2802] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 4 FOLLOWER]: Leader election won for term 4
I20250812 19:57:22.741511 17578 raft_consensus.cc:695] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 4 LEADER]: Becoming Leader. State: Replica: 7fb6a25b01a64037923c34f77051d99a, State: Running, Role: LEADER
I20250812 19:57:22.742343 17578 consensus_queue.cc:237] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true
I20250812 19:57:22.751039 17488 catalog_manager.cc:5582] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 7fb6a25b01a64037923c34f77051d99a (127.16.93.195), VOTER 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194). New cstate: current_term: 4 leader_uuid: "7fb6a25b01a64037923c34f77051d99a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfbad94c55478b9c448546d31813f8" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40943 } } peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true }
W20250812 19:57:23.197043 17257 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8ec0 after lost signal to thread 17132
W20250812 19:57:23.198004 17257 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd600 after lost signal to thread 17260
I20250812 19:57:23.247911 17082 raft_consensus.cc:1273] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Refusing update from remote peer 7fb6a25b01a64037923c34f77051d99a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250812 19:57:23.249366 17585 consensus_queue.cc:1035] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Connected to new peer: Peer: permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250812 19:57:23.259447 17588 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER cadfbad94c55478b9c448546d31813f8 (127.16.93.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true }
I20250812 19:57:23.261319 17082 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER cadfbad94c55478b9c448546d31813f8 (127.16.93.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } unsafe_config_change: true }
I20250812 19:57:23.298499 17488 catalog_manager.cc:5582] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a reported cstate change: config changed from index -1 to 3, VOTER cadfbad94c55478b9c448546d31813f8 (127.16.93.193) evicted, no longer has a pending config: VOTER 7fb6a25b01a64037923c34f77051d99a (127.16.93.195), VOTER 157206c188a14e3ba68afa99b2f7f127 (127.16.93.194). New cstate: current_term: 4 leader_uuid: "7fb6a25b01a64037923c34f77051d99a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250812 19:57:23.313884 17488 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 252544c7797f4f028ead00e021b1f280 on TS cadfbad94c55478b9c448546d31813f8: Not found: failed to reset TS proxy: Could not find TS for UUID cadfbad94c55478b9c448546d31813f8
I20250812 19:57:23.336813 17216 consensus_queue.cc:237] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: NON_VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: true } } unsafe_config_change: true
I20250812 19:57:23.344055 17082 raft_consensus.cc:1273] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Refusing update from remote peer 7fb6a25b01a64037923c34f77051d99a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250812 19:57:23.346011 17588 consensus_queue.cc:1035] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Connected to new peer: Peer: permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250812 19:57:23.354274 17585 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: NON_VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: true } } unsafe_config_change: true }
I20250812 19:57:23.362990 17082 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: NON_VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: true } } unsafe_config_change: true }
W20250812 19:57:23.370635 17473 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 252544c7797f4f028ead00e021b1f280 on TS cadfbad94c55478b9c448546d31813f8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID cadfbad94c55478b9c448546d31813f8
W20250812 19:57:23.375952 17152 consensus_peers.cc:489] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a -> Peer f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196:45543): Couldn't send request to peer f08f0f98d6b64950a82bf97f0935e4ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 252544c7797f4f028ead00e021b1f280. This is attempt 1: this message will repeat every 5th retry.
I20250812 19:57:23.389645 17475 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 252544c7797f4f028ead00e021b1f280 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250812 19:57:23.391709 17488 catalog_manager.cc:5582] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a reported cstate change: config changed from index 3 to 5, NON_VOTER f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) added. New cstate: current_term: 4 leader_uuid: "7fb6a25b01a64037923c34f77051d99a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: NON_VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
I20250812 19:57:23.884089 17599 ts_tablet_manager.cc:927] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Initiating tablet copy from peer 7fb6a25b01a64037923c34f77051d99a (127.16.93.195:39893)
I20250812 19:57:23.886163 17599 tablet_copy_client.cc:323] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: tablet copy: Beginning tablet copy session from remote peer at address 127.16.93.195:39893
I20250812 19:57:23.896911 17236 tablet_copy_service.cc:140] P 7fb6a25b01a64037923c34f77051d99a: Received BeginTabletCopySession request for tablet 252544c7797f4f028ead00e021b1f280 from peer f08f0f98d6b64950a82bf97f0935e4ea ({username='slave'} at 127.16.93.196:59119)
I20250812 19:57:23.897544 17236 tablet_copy_service.cc:161] P 7fb6a25b01a64037923c34f77051d99a: Beginning new tablet copy session on tablet 252544c7797f4f028ead00e021b1f280 from peer f08f0f98d6b64950a82bf97f0935e4ea at {username='slave'} at 127.16.93.196:59119: session id = f08f0f98d6b64950a82bf97f0935e4ea-252544c7797f4f028ead00e021b1f280
I20250812 19:57:23.906064 17236 tablet_copy_source_session.cc:215] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 19:57:23.911201 17599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252544c7797f4f028ead00e021b1f280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:23.928476 17599 tablet_copy_client.cc:806] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: tablet copy: Starting download of 0 data blocks...
I20250812 19:57:23.929088 17599 tablet_copy_client.cc:670] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: tablet copy: Starting download of 1 WAL segments...
I20250812 19:57:23.933115 17599 tablet_copy_client.cc:538] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 19:57:23.982065 17599 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Bootstrap starting.
I20250812 19:57:23.995021 17599 log.cc:826] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:24.009037 17599 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:57:24.009908 17599 tablet_bootstrap.cc:492] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Bootstrap complete.
I20250812 19:57:24.010515 17599 ts_tablet_manager.cc:1397] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20250812 19:57:24.029096 17599 raft_consensus.cc:357] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: NON_VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: true } } unsafe_config_change: true
I20250812 19:57:24.029917 17599 raft_consensus.cc:738] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: f08f0f98d6b64950a82bf97f0935e4ea, State: Initialized, Role: LEARNER
I20250812 19:57:24.030566 17599 consensus_queue.cc:260] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: NON_VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: true } } unsafe_config_change: true
I20250812 19:57:24.034739 17599 ts_tablet_manager.cc:1428] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea: Time spent starting tablet: real 0.024s user 0.024s sys 0.000s
I20250812 19:57:24.036568 17236 tablet_copy_service.cc:342] P 7fb6a25b01a64037923c34f77051d99a: Request end of tablet copy session f08f0f98d6b64950a82bf97f0935e4ea-252544c7797f4f028ead00e021b1f280 received from {username='slave'} at 127.16.93.196:59119
I20250812 19:57:24.037102 17236 tablet_copy_service.cc:434] P 7fb6a25b01a64037923c34f77051d99a: ending tablet copy session f08f0f98d6b64950a82bf97f0935e4ea-252544c7797f4f028ead00e021b1f280 on tablet 252544c7797f4f028ead00e021b1f280 with peer f08f0f98d6b64950a82bf97f0935e4ea
I20250812 19:57:24.491515 17350 raft_consensus.cc:1215] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250812 19:57:24.537694 17473 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 252544c7797f4f028ead00e021b1f280 on TS cadfbad94c55478b9c448546d31813f8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID cadfbad94c55478b9c448546d31813f8
I20250812 19:57:24.868757 17608 raft_consensus.cc:1062] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a: attempting to promote NON_VOTER f08f0f98d6b64950a82bf97f0935e4ea to VOTER
I20250812 19:57:24.870991 17608 consensus_queue.cc:237] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: false } } unsafe_config_change: true
I20250812 19:57:24.877646 17082 raft_consensus.cc:1273] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Refusing update from remote peer 7fb6a25b01a64037923c34f77051d99a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250812 19:57:24.877625 17350 raft_consensus.cc:1273] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea [term 4 LEARNER]: Refusing update from remote peer 7fb6a25b01a64037923c34f77051d99a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250812 19:57:24.879199 17608 consensus_queue.cc:1035] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Connected to new peer: Peer: permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250812 19:57:24.880204 17607 consensus_queue.cc:1035] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250812 19:57:24.897521 17606 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: false } } unsafe_config_change: true }
I20250812 19:57:24.899883 17350 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P f08f0f98d6b64950a82bf97f0935e4ea [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: false } } unsafe_config_change: true }
I20250812 19:57:24.902086 17082 raft_consensus.cc:2953] T 252544c7797f4f028ead00e021b1f280 P 157206c188a14e3ba68afa99b2f7f127 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: false } } unsafe_config_change: true }
I20250812 19:57:24.993127 17487 catalog_manager.cc:5582] T 252544c7797f4f028ead00e021b1f280 P 7fb6a25b01a64037923c34f77051d99a reported cstate change: config changed from index 5 to 6, f08f0f98d6b64950a82bf97f0935e4ea (127.16.93.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "7fb6a25b01a64037923c34f77051d99a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7fb6a25b01a64037923c34f77051d99a" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157206c188a14e3ba68afa99b2f7f127" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 35867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 45543 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250812 19:57:25.002624 16759 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250812 19:57:25.063433 16759 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "252544c7797f4f028ead00e021b1f280"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "7fb6a25b01a64037923c34f77051d99a"
rpc_addresses {
host: "127.16.93.195"
port: 39893
}
}
ts_infos {
permanent_uuid: "157206c188a14e3ba68afa99b2f7f127"
rpc_addresses {
host: "127.16.93.194"
port: 35867
}
}
ts_infos {
permanent_uuid: "f08f0f98d6b64950a82bf97f0935e4ea"
rpc_addresses {
host: "127.16.93.196"
port: 45543
}
}
I20250812 19:57:25.066382 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 16998
I20250812 19:57:25.091583 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17131
I20250812 19:57:25.123250 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17265
I20250812 19:57:25.148195 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17435
2025-08-12T19:57:25Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (21540 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250812 19:57:25.208899 16759 test_util.cc:276] Using random seed: 1928848166
I20250812 19:57:25.214609 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:57:25.214768 16759 ts_itest-base.cc:116] --------------
I20250812 19:57:25.214934 16759 ts_itest-base.cc:117] 3 tablet servers
I20250812 19:57:25.215083 16759 ts_itest-base.cc:118] 3 replicas per TS
I20250812 19:57:25.215236 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:57:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:57:25Z Disabled control of system clock
I20250812 19:57:25.250155 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:33125
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:44153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:33125
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250812 19:57:25.548350 17626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:25.548966 17626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:25.549423 17626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:25.578863 17626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:57:25.579207 17626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:25.579463 17626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:57:25.579699 17626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:57:25.614141 17626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:44153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:33125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:33125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:25.615468 17626 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:25.617060 17626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:25.627152 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:25.631980 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:25.628326 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:26.814384 17634 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1182 milliseconds
I20250812 19:57:26.814493 17626 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:26.815693 17626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:26.818775 17626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:26.820153 17626 hybrid_clock.cc:648] HybridClock initialized: now 1755028646820096 us; error 68 us; skew 500 ppm
I20250812 19:57:26.821091 17626 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:26.827001 17626 webserver.cc:480] Webserver started at http://127.16.93.254:39435/ using document root <none> and password file <none>
I20250812 19:57:26.827967 17626 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:26.828184 17626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:26.828688 17626 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:26.833202 17626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cbf695c39b1a4b069423a14c48cf72cb"
format_stamp: "Formatted at 2025-08-12 19:57:26 on dist-test-slave-wz96"
I20250812 19:57:26.834506 17626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cbf695c39b1a4b069423a14c48cf72cb"
format_stamp: "Formatted at 2025-08-12 19:57:26 on dist-test-slave-wz96"
I20250812 19:57:26.841542 17626 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250812 19:57:26.846925 17642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:26.848081 17626 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250812 19:57:26.848438 17626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cbf695c39b1a4b069423a14c48cf72cb"
format_stamp: "Formatted at 2025-08-12 19:57:26 on dist-test-slave-wz96"
I20250812 19:57:26.848840 17626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:26.902895 17626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:26.904397 17626 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:26.904852 17626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:26.973336 17626 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:33125
I20250812 19:57:26.973394 17693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:33125 every 8 connection(s)
I20250812 19:57:26.976042 17626 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:57:26.981077 17694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:26.984313 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17626
I20250812 19:57:26.984728 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:57:27.004884 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb: Bootstrap starting.
I20250812 19:57:27.010927 17694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:27.013067 17694 log.cc:826] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:27.017448 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb: No bootstrap required, opened a new log
I20250812 19:57:27.036667 17694 raft_consensus.cc:357] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf695c39b1a4b069423a14c48cf72cb" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33125 } }
I20250812 19:57:27.037705 17694 raft_consensus.cc:383] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:27.037963 17694 raft_consensus.cc:738] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf695c39b1a4b069423a14c48cf72cb, State: Initialized, Role: FOLLOWER
I20250812 19:57:27.038820 17694 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf695c39b1a4b069423a14c48cf72cb" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33125 } }
I20250812 19:57:27.039338 17694 raft_consensus.cc:397] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:57:27.039566 17694 raft_consensus.cc:491] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:57:27.039845 17694 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:27.044061 17694 raft_consensus.cc:513] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf695c39b1a4b069423a14c48cf72cb" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33125 } }
I20250812 19:57:27.044824 17694 leader_election.cc:304] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf695c39b1a4b069423a14c48cf72cb; no voters:
I20250812 19:57:27.046447 17694 leader_election.cc:290] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:57:27.047233 17699 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:27.049489 17699 raft_consensus.cc:695] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [term 1 LEADER]: Becoming Leader. State: Replica: cbf695c39b1a4b069423a14c48cf72cb, State: Running, Role: LEADER
I20250812 19:57:27.050316 17699 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf695c39b1a4b069423a14c48cf72cb" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33125 } }
I20250812 19:57:27.050657 17694 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:57:27.062134 17700 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf695c39b1a4b069423a14c48cf72cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf695c39b1a4b069423a14c48cf72cb" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33125 } } }
I20250812 19:57:27.062826 17700 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:27.063027 17701 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf695c39b1a4b069423a14c48cf72cb. Latest consensus state: current_term: 1 leader_uuid: "cbf695c39b1a4b069423a14c48cf72cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf695c39b1a4b069423a14c48cf72cb" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33125 } } }
I20250812 19:57:27.063673 17701 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:27.065549 17709 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:57:27.078454 17709 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:57:27.093966 17709 catalog_manager.cc:1349] Generated new cluster ID: 1528ed13771b4e988a2401161dff4fc5
I20250812 19:57:27.094236 17709 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:57:27.116137 17709 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:57:27.118145 17709 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:57:27.140403 17709 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cbf695c39b1a4b069423a14c48cf72cb: Generated new TSK 0
I20250812 19:57:27.141533 17709 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:57:27.166652 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33125
--builtin_ntp_servers=127.16.93.212:44153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250812 19:57:27.473055 17718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250812 19:57:27.473701 17718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:27.473959 17718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:27.474434 17718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:27.506032 17718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:27.506911 17718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:57:27.542753 17718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:44153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:27.544055 17718 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:27.545678 17718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:27.557935 17724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:27.562405 17727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:27.559860 17725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:29.058734 17726 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1495 milliseconds
I20250812 19:57:29.058881 17718 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:29.060236 17718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:29.063086 17718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:29.064546 17718 hybrid_clock.cc:648] HybridClock initialized: now 1755028649064504 us; error 66 us; skew 500 ppm
I20250812 19:57:29.065447 17718 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:29.073164 17718 webserver.cc:480] Webserver started at http://127.16.93.193:43051/ using document root <none> and password file <none>
I20250812 19:57:29.074095 17718 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:29.074282 17718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:29.074780 17718 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:29.079303 17718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "43ae5f60696241eea839967bc0a7643c"
format_stamp: "Formatted at 2025-08-12 19:57:29 on dist-test-slave-wz96"
I20250812 19:57:29.080613 17718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "43ae5f60696241eea839967bc0a7643c"
format_stamp: "Formatted at 2025-08-12 19:57:29 on dist-test-slave-wz96"
I20250812 19:57:29.088639 17718 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250812 19:57:29.094758 17734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:29.095917 17718 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250812 19:57:29.096252 17718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "43ae5f60696241eea839967bc0a7643c"
format_stamp: "Formatted at 2025-08-12 19:57:29 on dist-test-slave-wz96"
I20250812 19:57:29.096608 17718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:29.158561 17718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:29.159981 17718 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:29.160415 17718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:29.162992 17718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:29.167749 17718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:29.167966 17718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:29.168218 17718 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:29.168385 17718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:29.314689 17718 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:38193
I20250812 19:57:29.314785 17846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:38193 every 8 connection(s)
I20250812 19:57:29.317276 17718 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:57:29.327855 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17718
I20250812 19:57:29.328263 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:57:29.334383 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33125
--builtin_ntp_servers=127.16.93.212:44153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250812 19:57:29.338433 17847 heartbeater.cc:344] Connected to a master server at 127.16.93.254:33125
I20250812 19:57:29.338845 17847 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:29.339814 17847 heartbeater.cc:507] Master 127.16.93.254:33125 requested a full tablet report, sending...
I20250812 19:57:29.342252 17659 ts_manager.cc:194] Registered new tserver with Master: 43ae5f60696241eea839967bc0a7643c (127.16.93.193:38193)
I20250812 19:57:29.344089 17659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:37737
W20250812 19:57:29.637939 17851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250812 19:57:29.638561 17851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:29.638830 17851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:29.639297 17851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:29.670976 17851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:29.671854 17851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:57:29.708483 17851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:44153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:29.709836 17851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:29.711405 17851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:29.723133 17857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:30.347278 17847 heartbeater.cc:499] Master 127.16.93.254:33125 was elected leader, sending a full tablet report...
W20250812 19:57:29.723970 17858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:30.934772 17859 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1209 milliseconds
W20250812 19:57:30.935611 17860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:30.938059 17851 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.215s user 0.350s sys 0.850s
W20250812 19:57:30.938473 17851 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.215s user 0.350s sys 0.851s
I20250812 19:57:30.938894 17851 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:30.940418 17851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:30.943337 17851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:30.944895 17851 hybrid_clock.cc:648] HybridClock initialized: now 1755028650944840 us; error 45 us; skew 500 ppm
I20250812 19:57:30.946069 17851 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:30.954726 17851 webserver.cc:480] Webserver started at http://127.16.93.194:38283/ using document root <none> and password file <none>
I20250812 19:57:30.956147 17851 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:30.956475 17851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:30.957142 17851 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:30.964691 17851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9420a057ee934457bacf0c059e12ea94"
format_stamp: "Formatted at 2025-08-12 19:57:30 on dist-test-slave-wz96"
I20250812 19:57:30.966346 17851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9420a057ee934457bacf0c059e12ea94"
format_stamp: "Formatted at 2025-08-12 19:57:30 on dist-test-slave-wz96"
I20250812 19:57:30.976809 17851 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.002s
I20250812 19:57:30.984525 17867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:30.985852 17851 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20250812 19:57:30.986218 17851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9420a057ee934457bacf0c059e12ea94"
format_stamp: "Formatted at 2025-08-12 19:57:30 on dist-test-slave-wz96"
I20250812 19:57:30.986657 17851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:31.059711 17851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:31.061241 17851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:31.061672 17851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:31.064082 17851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:31.068228 17851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:31.068425 17851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:31.068732 17851 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:31.068884 17851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:31.197911 17851 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:39649
I20250812 19:57:31.198009 17979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:39649 every 8 connection(s)
I20250812 19:57:31.200444 17851 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:57:31.206254 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17851
I20250812 19:57:31.206825 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 19:57:31.213358 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33125
--builtin_ntp_servers=127.16.93.212:44153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250812 19:57:31.222604 17980 heartbeater.cc:344] Connected to a master server at 127.16.93.254:33125
I20250812 19:57:31.223146 17980 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:31.224471 17980 heartbeater.cc:507] Master 127.16.93.254:33125 requested a full tablet report, sending...
I20250812 19:57:31.226872 17659 ts_manager.cc:194] Registered new tserver with Master: 9420a057ee934457bacf0c059e12ea94 (127.16.93.194:39649)
I20250812 19:57:31.228085 17659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:35389
W20250812 19:57:31.516265 17984 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250812 19:57:31.516903 17984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:31.517179 17984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:31.517655 17984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:31.548695 17984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:31.549554 17984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:57:31.583952 17984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:44153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:31.585250 17984 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:31.586769 17984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:31.598079 17990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:32.232143 17980 heartbeater.cc:499] Master 127.16.93.254:33125 was elected leader, sending a full tablet report...
W20250812 19:57:31.598634 17991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:32.889880 17993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:32.891686 17984 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.293s user 0.468s sys 0.810s
W20250812 19:57:32.891959 17984 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.294s user 0.468s sys 0.810s
W20250812 19:57:32.892647 17992 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1288 milliseconds
I20250812 19:57:32.892693 17984 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:32.893932 17984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:32.905174 17984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:32.906620 17984 hybrid_clock.cc:648] HybridClock initialized: now 1755028652906573 us; error 71 us; skew 500 ppm
I20250812 19:57:32.907433 17984 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:32.914412 17984 webserver.cc:480] Webserver started at http://127.16.93.195:45657/ using document root <none> and password file <none>
I20250812 19:57:32.915333 17984 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:32.915547 17984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:32.915987 17984 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:32.920707 17984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "51ea7e4d882847b3b92fa0ecb9eed107"
format_stamp: "Formatted at 2025-08-12 19:57:32 on dist-test-slave-wz96"
I20250812 19:57:32.921803 17984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "51ea7e4d882847b3b92fa0ecb9eed107"
format_stamp: "Formatted at 2025-08-12 19:57:32 on dist-test-slave-wz96"
I20250812 19:57:32.929699 17984 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20250812 19:57:32.935951 18000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:32.937266 17984 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 19:57:32.937606 17984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "51ea7e4d882847b3b92fa0ecb9eed107"
format_stamp: "Formatted at 2025-08-12 19:57:32 on dist-test-slave-wz96"
I20250812 19:57:32.937973 17984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:33.017771 17984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:33.019186 17984 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:33.019625 17984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:33.022109 17984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:33.026170 17984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:33.026389 17984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:33.026762 17984 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:33.026928 17984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:33.157706 17984 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:32809
I20250812 19:57:33.157815 18112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:32809 every 8 connection(s)
I20250812 19:57:33.160259 17984 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:57:33.169907 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 17984
I20250812 19:57:33.170423 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 19:57:33.186954 18113 heartbeater.cc:344] Connected to a master server at 127.16.93.254:33125
I20250812 19:57:33.187438 18113 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:33.188606 18113 heartbeater.cc:507] Master 127.16.93.254:33125 requested a full tablet report, sending...
I20250812 19:57:33.191094 17658 ts_manager.cc:194] Registered new tserver with Master: 51ea7e4d882847b3b92fa0ecb9eed107 (127.16.93.195:32809)
I20250812 19:57:33.192404 17658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:38611
I20250812 19:57:33.205093 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:57:33.236126 17658 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49472:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250812 19:57:33.255210 17658 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 19:57:33.310237 17915 tablet_service.cc:1468] Processing CreateTablet for tablet d3697da0430640c0a58bdf092f0c6a21 (DEFAULT_TABLE table=TestTable [id=672f00df3e1643e38a78b3076558f96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:33.310724 17782 tablet_service.cc:1468] Processing CreateTablet for tablet d3697da0430640c0a58bdf092f0c6a21 (DEFAULT_TABLE table=TestTable [id=672f00df3e1643e38a78b3076558f96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:33.311888 17915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3697da0430640c0a58bdf092f0c6a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:33.312567 17782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3697da0430640c0a58bdf092f0c6a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:33.313740 18048 tablet_service.cc:1468] Processing CreateTablet for tablet d3697da0430640c0a58bdf092f0c6a21 (DEFAULT_TABLE table=TestTable [id=672f00df3e1643e38a78b3076558f96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:33.315611 18048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3697da0430640c0a58bdf092f0c6a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:33.345875 18132 tablet_bootstrap.cc:492] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: Bootstrap starting.
I20250812 19:57:33.346375 18133 tablet_bootstrap.cc:492] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: Bootstrap starting.
I20250812 19:57:33.352787 18132 tablet_bootstrap.cc:654] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:33.354537 18132 log.cc:826] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:33.354869 18133 tablet_bootstrap.cc:654] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:33.357281 18133 log.cc:826] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:33.362491 18135 tablet_bootstrap.cc:492] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: Bootstrap starting.
I20250812 19:57:33.364966 18133 tablet_bootstrap.cc:492] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: No bootstrap required, opened a new log
I20250812 19:57:33.365399 18133 ts_tablet_manager.cc:1397] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20250812 19:57:33.365891 18132 tablet_bootstrap.cc:492] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: No bootstrap required, opened a new log
I20250812 19:57:33.366393 18132 ts_tablet_manager.cc:1397] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20250812 19:57:33.373093 18135 tablet_bootstrap.cc:654] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:33.375566 18135 log.cc:826] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:33.381436 18135 tablet_bootstrap.cc:492] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: No bootstrap required, opened a new log
I20250812 19:57:33.381960 18135 ts_tablet_manager.cc:1397] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20250812 19:57:33.384793 18133 raft_consensus.cc:357] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.385469 18133 raft_consensus.cc:738] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ae5f60696241eea839967bc0a7643c, State: Initialized, Role: FOLLOWER
I20250812 19:57:33.386049 18133 consensus_queue.cc:260] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.394651 18133 ts_tablet_manager.cc:1428] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: Time spent starting tablet: real 0.029s user 0.020s sys 0.008s
I20250812 19:57:33.393463 18132 raft_consensus.cc:357] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.395620 18132 raft_consensus.cc:738] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9420a057ee934457bacf0c059e12ea94, State: Initialized, Role: FOLLOWER
I20250812 19:57:33.396301 18132 consensus_queue.cc:260] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.402889 18132 ts_tablet_manager.cc:1428] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: Time spent starting tablet: real 0.036s user 0.025s sys 0.008s
I20250812 19:57:33.407372 18135 raft_consensus.cc:357] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.408059 18135 raft_consensus.cc:738] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ea7e4d882847b3b92fa0ecb9eed107, State: Initialized, Role: FOLLOWER
I20250812 19:57:33.408847 18135 consensus_queue.cc:260] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.415776 18113 heartbeater.cc:499] Master 127.16.93.254:33125 was elected leader, sending a full tablet report...
I20250812 19:57:33.416405 18135 ts_tablet_manager.cc:1428] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: Time spent starting tablet: real 0.034s user 0.030s sys 0.004s
I20250812 19:57:33.422648 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:57:33.426641 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 43ae5f60696241eea839967bc0a7643c to finish bootstrapping
I20250812 19:57:33.441108 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9420a057ee934457bacf0c059e12ea94 to finish bootstrapping
I20250812 19:57:33.452994 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 51ea7e4d882847b3b92fa0ecb9eed107 to finish bootstrapping
W20250812 19:57:33.456892 17981 tablet.cc:2378] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:33.507817 17802 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21"
dest_uuid: "43ae5f60696241eea839967bc0a7643c"
from {username='slave'} at 127.0.0.1:36728
I20250812 19:57:33.508406 17802 raft_consensus.cc:491] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250812 19:57:33.508783 17802 raft_consensus.cc:3058] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:33.513496 17802 raft_consensus.cc:513] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.515967 17802 leader_election.cc:290] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [CANDIDATE]: Term 1 election: Requested vote from peers 9420a057ee934457bacf0c059e12ea94 (127.16.93.194:39649), 51ea7e4d882847b3b92fa0ecb9eed107 (127.16.93.195:32809)
I20250812 19:57:33.524961 16759 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250812 19:57:33.528766 17935 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21" candidate_uuid: "43ae5f60696241eea839967bc0a7643c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9420a057ee934457bacf0c059e12ea94"
I20250812 19:57:33.529400 17935 raft_consensus.cc:3058] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:33.530396 18068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21" candidate_uuid: "43ae5f60696241eea839967bc0a7643c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51ea7e4d882847b3b92fa0ecb9eed107"
I20250812 19:57:33.531041 18068 raft_consensus.cc:3058] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:33.533924 17935 raft_consensus.cc:2466] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ae5f60696241eea839967bc0a7643c in term 1.
I20250812 19:57:33.535033 17735 leader_election.cc:304] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ae5f60696241eea839967bc0a7643c, 9420a057ee934457bacf0c059e12ea94; no voters:
I20250812 19:57:33.535452 18068 raft_consensus.cc:2466] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ae5f60696241eea839967bc0a7643c in term 1.
I20250812 19:57:33.535858 18138 raft_consensus.cc:2802] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:33.537655 18138 raft_consensus.cc:695] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 LEADER]: Becoming Leader. State: Replica: 43ae5f60696241eea839967bc0a7643c, State: Running, Role: LEADER
I20250812 19:57:33.538532 18138 consensus_queue.cc:237] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:33.549510 17658 catalog_manager.cc:5582] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c reported cstate change: term changed from 0 to 1, leader changed from <none> to 43ae5f60696241eea839967bc0a7643c (127.16.93.193). New cstate: current_term: 1 leader_uuid: "43ae5f60696241eea839967bc0a7643c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } health_report { overall_health: HEALTHY } } }
W20250812 19:57:33.576431 17848 tablet.cc:2378] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:33.630651 16759 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250812 19:57:33.664959 18114 tablet.cc:2378] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:33.836503 16759 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250812 19:57:33.995563 18138 consensus_queue.cc:1035] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 19:57:34.013316 18152 consensus_queue.cc:1035] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250812 19:57:35.859380 17802 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21"
dest_uuid: "43ae5f60696241eea839967bc0a7643c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34374
I20250812 19:57:35.859867 17802 raft_consensus.cc:604] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 LEADER]: Received request to transfer leadership
I20250812 19:57:36.073046 18171 raft_consensus.cc:991] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c: : Instructing follower 51ea7e4d882847b3b92fa0ecb9eed107 to start an election
I20250812 19:57:36.073418 18171 raft_consensus.cc:1079] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 LEADER]: Signalling peer 51ea7e4d882847b3b92fa0ecb9eed107 to start an election
I20250812 19:57:36.074832 18068 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21"
dest_uuid: "51ea7e4d882847b3b92fa0ecb9eed107"
from {username='slave'} at 127.16.93.193:46449
I20250812 19:57:36.075356 18068 raft_consensus.cc:491] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250812 19:57:36.075625 18068 raft_consensus.cc:3058] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:57:36.080140 18068 raft_consensus.cc:513] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:36.082494 18068 leader_election.cc:290] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [CANDIDATE]: Term 2 election: Requested vote from peers 9420a057ee934457bacf0c059e12ea94 (127.16.93.194:39649), 43ae5f60696241eea839967bc0a7643c (127.16.93.193:38193)
I20250812 19:57:36.094722 17935 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21" candidate_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9420a057ee934457bacf0c059e12ea94"
I20250812 19:57:36.094781 17802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21" candidate_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "43ae5f60696241eea839967bc0a7643c"
I20250812 19:57:36.095364 17935 raft_consensus.cc:3058] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:57:36.095387 17802 raft_consensus.cc:3053] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 LEADER]: Stepping down as leader of term 1
I20250812 19:57:36.095702 17802 raft_consensus.cc:738] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43ae5f60696241eea839967bc0a7643c, State: Running, Role: LEADER
I20250812 19:57:36.096421 17802 consensus_queue.cc:260] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:36.097720 17802 raft_consensus.cc:3058] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:57:36.101945 17935 raft_consensus.cc:2466] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ea7e4d882847b3b92fa0ecb9eed107 in term 2.
I20250812 19:57:36.103250 18001 leader_election.cc:304] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ea7e4d882847b3b92fa0ecb9eed107, 9420a057ee934457bacf0c059e12ea94; no voters:
I20250812 19:57:36.104207 17802 raft_consensus.cc:2466] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ea7e4d882847b3b92fa0ecb9eed107 in term 2.
I20250812 19:57:36.105427 18186 raft_consensus.cc:2802] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 19:57:36.106724 18186 raft_consensus.cc:695] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [term 2 LEADER]: Becoming Leader. State: Replica: 51ea7e4d882847b3b92fa0ecb9eed107, State: Running, Role: LEADER
I20250812 19:57:36.107554 18186 consensus_queue.cc:237] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } }
I20250812 19:57:36.114748 17658 catalog_manager.cc:5582] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 reported cstate change: term changed from 1 to 2, leader changed from 43ae5f60696241eea839967bc0a7643c (127.16.93.193) to 51ea7e4d882847b3b92fa0ecb9eed107 (127.16.93.195). New cstate: current_term: 2 leader_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51ea7e4d882847b3b92fa0ecb9eed107" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 } health_report { overall_health: UNKNOWN } } }
I20250812 19:57:36.618868 17802 raft_consensus.cc:1273] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 2 FOLLOWER]: Refusing update from remote peer 51ea7e4d882847b3b92fa0ecb9eed107: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250812 19:57:36.620167 18193 consensus_queue.cc:1035] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ae5f60696241eea839967bc0a7643c" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250812 19:57:36.630810 17935 raft_consensus.cc:1273] T d3697da0430640c0a58bdf092f0c6a21 P 9420a057ee934457bacf0c059e12ea94 [term 2 FOLLOWER]: Refusing update from remote peer 51ea7e4d882847b3b92fa0ecb9eed107: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250812 19:57:36.632746 18193 consensus_queue.cc:1035] T d3697da0430640c0a58bdf092f0c6a21 P 51ea7e4d882847b3b92fa0ecb9eed107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9420a057ee934457bacf0c059e12ea94" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250812 19:57:38.569685 17802 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d3697da0430640c0a58bdf092f0c6a21"
dest_uuid: "43ae5f60696241eea839967bc0a7643c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34376
I20250812 19:57:38.570494 17802 raft_consensus.cc:604] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 2 FOLLOWER]: Received request to transfer leadership
I20250812 19:57:38.570917 17802 raft_consensus.cc:612] T d3697da0430640c0a58bdf092f0c6a21 P 43ae5f60696241eea839967bc0a7643c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250812 19:57:39.609267 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17718
I20250812 19:57:39.635699 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17851
I20250812 19:57:39.658757 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17984
I20250812 19:57:39.683840 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 17626
2025-08-12T19:57:39Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14531 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250812 19:57:39.740548 16759 test_util.cc:276] Using random seed: 1943379814
I20250812 19:57:39.744721 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:57:39.744899 16759 ts_itest-base.cc:116] --------------
I20250812 19:57:39.745062 16759 ts_itest-base.cc:117] 3 tablet servers
I20250812 19:57:39.745201 16759 ts_itest-base.cc:118] 3 replicas per TS
I20250812 19:57:39.745342 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:57:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:57:39Z Disabled control of system clock
I20250812 19:57:39.780694 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:43435
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:43435 with env {}
W20250812 19:57:40.076309 18237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:40.076898 18237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:40.077332 18237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:40.107630 18237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:57:40.107918 18237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:40.108129 18237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:57:40.108323 18237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:57:40.142868 18237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:43435
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:40.144090 18237 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:40.145610 18237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:40.155071 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:40.156905 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:41.353117 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:41.355192 18245 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
I20250812 19:57:41.355286 18237 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:41.356539 18237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:41.359575 18237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:41.361021 18237 hybrid_clock.cc:648] HybridClock initialized: now 1755028661360982 us; error 53 us; skew 500 ppm
I20250812 19:57:41.361814 18237 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:41.368137 18237 webserver.cc:480] Webserver started at http://127.16.93.254:32839/ using document root <none> and password file <none>
I20250812 19:57:41.369062 18237 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:41.369305 18237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:41.369757 18237 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:41.374127 18237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d872bfa155894b559c0d8283a36a5b1d"
format_stamp: "Formatted at 2025-08-12 19:57:41 on dist-test-slave-wz96"
I20250812 19:57:41.375162 18237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d872bfa155894b559c0d8283a36a5b1d"
format_stamp: "Formatted at 2025-08-12 19:57:41 on dist-test-slave-wz96"
I20250812 19:57:41.382030 18237 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250812 19:57:41.387295 18253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:41.388386 18237 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250812 19:57:41.388744 18237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d872bfa155894b559c0d8283a36a5b1d"
format_stamp: "Formatted at 2025-08-12 19:57:41 on dist-test-slave-wz96"
I20250812 19:57:41.389068 18237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:41.439220 18237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:41.441006 18237 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:41.441447 18237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:41.509703 18237 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:43435
I20250812 19:57:41.509764 18304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:43435 every 8 connection(s)
I20250812 19:57:41.512411 18237 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:57:41.516811 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 18237
I20250812 19:57:41.517177 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:57:41.517436 18305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:41.540460 18305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d: Bootstrap starting.
I20250812 19:57:41.545826 18305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:41.547676 18305 log.cc:826] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:41.551903 18305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d: No bootstrap required, opened a new log
I20250812 19:57:41.568657 18305 raft_consensus.cc:357] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d872bfa155894b559c0d8283a36a5b1d" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 43435 } }
I20250812 19:57:41.569545 18305 raft_consensus.cc:383] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:41.569766 18305 raft_consensus.cc:738] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d872bfa155894b559c0d8283a36a5b1d, State: Initialized, Role: FOLLOWER
I20250812 19:57:41.570539 18305 consensus_queue.cc:260] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d872bfa155894b559c0d8283a36a5b1d" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 43435 } }
I20250812 19:57:41.571030 18305 raft_consensus.cc:397] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:57:41.571305 18305 raft_consensus.cc:491] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:57:41.571600 18305 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:41.575423 18305 raft_consensus.cc:513] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d872bfa155894b559c0d8283a36a5b1d" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 43435 } }
I20250812 19:57:41.576112 18305 leader_election.cc:304] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d872bfa155894b559c0d8283a36a5b1d; no voters:
I20250812 19:57:41.577754 18305 leader_election.cc:290] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:57:41.578536 18310 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:41.580796 18310 raft_consensus.cc:695] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [term 1 LEADER]: Becoming Leader. State: Replica: d872bfa155894b559c0d8283a36a5b1d, State: Running, Role: LEADER
I20250812 19:57:41.581607 18310 consensus_queue.cc:237] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d872bfa155894b559c0d8283a36a5b1d" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 43435 } }
I20250812 19:57:41.582064 18305 sys_catalog.cc:564] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:57:41.593272 18312 sys_catalog.cc:455] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d872bfa155894b559c0d8283a36a5b1d. Latest consensus state: current_term: 1 leader_uuid: "d872bfa155894b559c0d8283a36a5b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d872bfa155894b559c0d8283a36a5b1d" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 43435 } } }
I20250812 19:57:41.594408 18312 sys_catalog.cc:458] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:41.596086 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d872bfa155894b559c0d8283a36a5b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d872bfa155894b559c0d8283a36a5b1d" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 43435 } } }
I20250812 19:57:41.596923 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d [sys.catalog]: This master's current role is: LEADER
I20250812 19:57:41.598549 18319 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:57:41.609440 18319 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:57:41.624301 18319 catalog_manager.cc:1349] Generated new cluster ID: 5fdf77dcb1004cc180bb474327c7e4aa
I20250812 19:57:41.624650 18319 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:57:41.653980 18319 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:57:41.655977 18319 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:57:41.677578 18319 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d872bfa155894b559c0d8283a36a5b1d: Generated new TSK 0
I20250812 19:57:41.678485 18319 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:57:41.688455 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:43435
--builtin_ntp_servers=127.16.93.212:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 19:57:41.994453 18329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:41.994977 18329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:41.995488 18329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:42.027182 18329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:42.028038 18329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:57:42.063850 18329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:43435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:42.065245 18329 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:42.066824 18329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:42.080207 18335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:42.083985 18338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:42.082432 18336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:43.589044 18337 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1504 milliseconds
I20250812 19:57:43.589170 18329 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:43.590454 18329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:43.593227 18329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:43.594717 18329 hybrid_clock.cc:648] HybridClock initialized: now 1755028663594657 us; error 78 us; skew 500 ppm
I20250812 19:57:43.595590 18329 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:43.602756 18329 webserver.cc:480] Webserver started at http://127.16.93.193:38833/ using document root <none> and password file <none>
I20250812 19:57:43.603727 18329 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:43.603929 18329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:43.604435 18329 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:43.608999 18329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7df4d82dafc64d678150162ecb4cb30f"
format_stamp: "Formatted at 2025-08-12 19:57:43 on dist-test-slave-wz96"
I20250812 19:57:43.610203 18329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7df4d82dafc64d678150162ecb4cb30f"
format_stamp: "Formatted at 2025-08-12 19:57:43 on dist-test-slave-wz96"
I20250812 19:57:43.617973 18329 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250812 19:57:43.624265 18345 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:43.625453 18329 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 19:57:43.625794 18329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7df4d82dafc64d678150162ecb4cb30f"
format_stamp: "Formatted at 2025-08-12 19:57:43 on dist-test-slave-wz96"
I20250812 19:57:43.626129 18329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:43.680492 18329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:43.682003 18329 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:43.682461 18329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:43.685196 18329 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:43.689906 18329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:43.690147 18329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:43.690402 18329 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:43.690568 18329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:43.857877 18329 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:33997
I20250812 19:57:43.857987 18458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:33997 every 8 connection(s)
I20250812 19:57:43.860706 18329 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:57:43.864174 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 18329
I20250812 19:57:43.864902 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:57:43.873791 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:43435
--builtin_ntp_servers=127.16.93.212:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:57:43.887727 18459 heartbeater.cc:344] Connected to a master server at 127.16.93.254:43435
I20250812 19:57:43.888274 18459 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:43.889454 18459 heartbeater.cc:507] Master 127.16.93.254:43435 requested a full tablet report, sending...
I20250812 19:57:43.891986 18270 ts_manager.cc:194] Registered new tserver with Master: 7df4d82dafc64d678150162ecb4cb30f (127.16.93.193:33997)
I20250812 19:57:43.894135 18270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:35453
W20250812 19:57:44.193060 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:44.193552 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:44.193996 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:44.226603 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:44.227416 18463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:57:44.263280 18463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:43435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:44.264542 18463 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:44.266175 18463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:44.278034 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:44.897742 18459 heartbeater.cc:499] Master 127.16.93.254:43435 was elected leader, sending a full tablet report...
W20250812 19:57:45.682050 18468 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 18463
W20250812 19:57:46.033155 18468 kernel_stack_watchdog.cc:198] Thread 18463 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:57:44.278908 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:46.034432 18463 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.756s user 0.687s sys 1.070s
W20250812 19:57:46.034844 18463 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.757s user 0.687s sys 1.070s
W20250812 19:57:46.036541 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:46.038625 18471 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1759 milliseconds
I20250812 19:57:46.038688 18463 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:46.040019 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:46.042062 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:46.043402 18463 hybrid_clock.cc:648] HybridClock initialized: now 1755028666043347 us; error 61 us; skew 500 ppm
I20250812 19:57:46.044229 18463 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:46.050168 18463 webserver.cc:480] Webserver started at http://127.16.93.194:35873/ using document root <none> and password file <none>
I20250812 19:57:46.051093 18463 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:46.051311 18463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:46.051755 18463 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:46.056198 18463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b"
format_stamp: "Formatted at 2025-08-12 19:57:46 on dist-test-slave-wz96"
I20250812 19:57:46.057327 18463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b"
format_stamp: "Formatted at 2025-08-12 19:57:46 on dist-test-slave-wz96"
I20250812 19:57:46.064234 18463 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250812 19:57:46.069629 18479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:46.070609 18463 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250812 19:57:46.070924 18463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b"
format_stamp: "Formatted at 2025-08-12 19:57:46 on dist-test-slave-wz96"
I20250812 19:57:46.071249 18463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:46.119449 18463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:46.120977 18463 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:46.121506 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:46.124135 18463 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:46.128288 18463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:46.128506 18463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:46.128815 18463 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:46.128980 18463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:46.259644 18463 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:36275
I20250812 19:57:46.259814 18591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:36275 every 8 connection(s)
I20250812 19:57:46.262171 18463 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:57:46.269446 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 18463
I20250812 19:57:46.269933 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 19:57:46.276166 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:43435
--builtin_ntp_servers=127.16.93.212:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:57:46.283882 18592 heartbeater.cc:344] Connected to a master server at 127.16.93.254:43435
I20250812 19:57:46.284425 18592 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:46.285735 18592 heartbeater.cc:507] Master 127.16.93.254:43435 requested a full tablet report, sending...
I20250812 19:57:46.288303 18270 ts_manager.cc:194] Registered new tserver with Master: a1b8d42ac5424bbe82ef1a3baa7f5f5b (127.16.93.194:36275)
I20250812 19:57:46.289580 18270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:39211
W20250812 19:57:46.575191 18596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:46.575712 18596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:57:46.576218 18596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:46.607280 18596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:57:46.608163 18596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:57:46.643139 18596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:43435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:57:46.644415 18596 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:57:46.645946 18596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:57:46.658250 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:57:47.293504 18592 heartbeater.cc:499] Master 127.16.93.254:43435 was elected leader, sending a full tablet report...
W20250812 19:57:48.061766 18601 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 18596
W20250812 19:57:48.388146 18596 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.730s user 0.614s sys 1.112s
W20250812 19:57:48.388502 18596 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.731s user 0.614s sys 1.112s
W20250812 19:57:46.658779 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:48.390323 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:57:48.392818 18604 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1731 milliseconds
I20250812 19:57:48.392844 18596 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:57:48.394119 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:57:48.396080 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:57:48.397396 18596 hybrid_clock.cc:648] HybridClock initialized: now 1755028668397356 us; error 49 us; skew 500 ppm
I20250812 19:57:48.398130 18596 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:57:48.403867 18596 webserver.cc:480] Webserver started at http://127.16.93.195:41365/ using document root <none> and password file <none>
I20250812 19:57:48.404799 18596 fs_manager.cc:362] Metadata directory not provided
I20250812 19:57:48.405089 18596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:57:48.405519 18596 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:57:48.409945 18596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "62fbef1b58b946b1977e77b39dfbaf62"
format_stamp: "Formatted at 2025-08-12 19:57:48 on dist-test-slave-wz96"
I20250812 19:57:48.411057 18596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "62fbef1b58b946b1977e77b39dfbaf62"
format_stamp: "Formatted at 2025-08-12 19:57:48 on dist-test-slave-wz96"
I20250812 19:57:48.418046 18596 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250812 19:57:48.423496 18612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:48.424651 18596 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 19:57:48.424973 18596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "62fbef1b58b946b1977e77b39dfbaf62"
format_stamp: "Formatted at 2025-08-12 19:57:48 on dist-test-slave-wz96"
I20250812 19:57:48.425307 18596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:57:48.527770 18596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:57:48.529268 18596 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:57:48.529716 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:57:48.532160 18596 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:57:48.536367 18596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:57:48.536612 18596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:48.536885 18596 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:57:48.537063 18596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:57:48.668838 18596 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:46389
I20250812 19:57:48.668948 18724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:46389 every 8 connection(s)
I20250812 19:57:48.671455 18596 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:57:48.673254 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 18596
I20250812 19:57:48.673776 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 19:57:48.691891 18725 heartbeater.cc:344] Connected to a master server at 127.16.93.254:43435
I20250812 19:57:48.692420 18725 heartbeater.cc:461] Registering TS with master...
I20250812 19:57:48.693572 18725 heartbeater.cc:507] Master 127.16.93.254:43435 requested a full tablet report, sending...
I20250812 19:57:48.695612 18269 ts_manager.cc:194] Registered new tserver with Master: 62fbef1b58b946b1977e77b39dfbaf62 (127.16.93.195:46389)
I20250812 19:57:48.696918 18269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:50207
I20250812 19:57:48.708215 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:57:48.742375 18269 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52554:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250812 19:57:48.761010 18269 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 19:57:48.830786 18660 tablet_service.cc:1468] Processing CreateTablet for tablet d284e91775c84020929b4378f7af93c1 (DEFAULT_TABLE table=TestTable [id=0b857ae357164006a81310fab44ebe12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:48.834152 18660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d284e91775c84020929b4378f7af93c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:48.834199 18527 tablet_service.cc:1468] Processing CreateTablet for tablet d284e91775c84020929b4378f7af93c1 (DEFAULT_TABLE table=TestTable [id=0b857ae357164006a81310fab44ebe12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:48.833458 18394 tablet_service.cc:1468] Processing CreateTablet for tablet d284e91775c84020929b4378f7af93c1 (DEFAULT_TABLE table=TestTable [id=0b857ae357164006a81310fab44ebe12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:57:48.836143 18527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d284e91775c84020929b4378f7af93c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:48.836148 18394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d284e91775c84020929b4378f7af93c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:48.855163 18744 tablet_bootstrap.cc:492] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: Bootstrap starting.
I20250812 19:57:48.864328 18744 tablet_bootstrap.cc:654] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:48.867514 18744 log.cc:826] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:48.877163 18745 tablet_bootstrap.cc:492] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: Bootstrap starting.
I20250812 19:57:48.878504 18746 tablet_bootstrap.cc:492] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Bootstrap starting.
I20250812 19:57:48.886368 18745 tablet_bootstrap.cc:654] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:48.886368 18746 tablet_bootstrap.cc:654] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:48.887229 18744 tablet_bootstrap.cc:492] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: No bootstrap required, opened a new log
I20250812 19:57:48.887789 18744 ts_tablet_manager.cc:1397] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.016s
I20250812 19:57:48.888907 18746 log.cc:826] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:48.892247 18745 log.cc:826] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: Log is configured to *not* fsync() on all Append() calls
I20250812 19:57:48.901153 18746 tablet_bootstrap.cc:492] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: No bootstrap required, opened a new log
I20250812 19:57:48.901690 18746 ts_tablet_manager.cc:1397] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.014s
I20250812 19:57:48.902032 18745 tablet_bootstrap.cc:492] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: No bootstrap required, opened a new log
I20250812 19:57:48.902547 18745 ts_tablet_manager.cc:1397] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.018s
I20250812 19:57:48.909348 18744 raft_consensus.cc:357] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:48.910024 18744 raft_consensus.cc:383] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:48.910315 18744 raft_consensus.cc:738] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fbef1b58b946b1977e77b39dfbaf62, State: Initialized, Role: FOLLOWER
I20250812 19:57:48.911267 18744 consensus_queue.cc:260] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:48.914446 18725 heartbeater.cc:499] Master 127.16.93.254:43435 was elected leader, sending a full tablet report...
I20250812 19:57:48.915903 18744 ts_tablet_manager.cc:1428] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: Time spent starting tablet: real 0.028s user 0.016s sys 0.010s
W20250812 19:57:48.926594 18726 tablet.cc:2378] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:48.927315 18746 raft_consensus.cc:357] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:48.928123 18746 raft_consensus.cc:383] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:48.928380 18746 raft_consensus.cc:738] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b8d42ac5424bbe82ef1a3baa7f5f5b, State: Initialized, Role: FOLLOWER
I20250812 19:57:48.929162 18746 consensus_queue.cc:260] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:48.929543 18745 raft_consensus.cc:357] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:48.930452 18745 raft_consensus.cc:383] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:48.930692 18745 raft_consensus.cc:738] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df4d82dafc64d678150162ecb4cb30f, State: Initialized, Role: FOLLOWER
I20250812 19:57:48.931805 18745 consensus_queue.cc:260] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:48.933009 18746 ts_tablet_manager.cc:1428] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Time spent starting tablet: real 0.031s user 0.024s sys 0.008s
I20250812 19:57:48.935693 18745 ts_tablet_manager.cc:1428] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: Time spent starting tablet: real 0.033s user 0.024s sys 0.007s
W20250812 19:57:49.019973 18593 tablet.cc:2378] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:49.090209 18752 raft_consensus.cc:491] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:57:49.090684 18752 raft_consensus.cc:513] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:49.093029 18752 leader_election.cc:290] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1b8d42ac5424bbe82ef1a3baa7f5f5b (127.16.93.194:36275), 62fbef1b58b946b1977e77b39dfbaf62 (127.16.93.195:46389)
I20250812 19:57:49.104130 18547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d284e91775c84020929b4378f7af93c1" candidate_uuid: "7df4d82dafc64d678150162ecb4cb30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" is_pre_election: true
I20250812 19:57:49.104481 18680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d284e91775c84020929b4378f7af93c1" candidate_uuid: "7df4d82dafc64d678150162ecb4cb30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fbef1b58b946b1977e77b39dfbaf62" is_pre_election: true
I20250812 19:57:49.104831 18751 raft_consensus.cc:491] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:57:49.105366 18680 raft_consensus.cc:2466] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df4d82dafc64d678150162ecb4cb30f in term 0.
I20250812 19:57:49.105356 18751 raft_consensus.cc:513] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:49.106352 18547 raft_consensus.cc:2466] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df4d82dafc64d678150162ecb4cb30f in term 0.
I20250812 19:57:49.106549 18346 leader_election.cc:304] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fbef1b58b946b1977e77b39dfbaf62, 7df4d82dafc64d678150162ecb4cb30f; no voters:
I20250812 19:57:49.107414 18752 raft_consensus.cc:2802] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:57:49.107780 18752 raft_consensus.cc:491] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:57:49.108110 18752 raft_consensus.cc:3058] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:49.110221 18751 leader_election.cc:290] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62fbef1b58b946b1977e77b39dfbaf62 (127.16.93.195:46389), 7df4d82dafc64d678150162ecb4cb30f (127.16.93.193:33997)
I20250812 19:57:49.116086 18752 raft_consensus.cc:513] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:49.117726 18752 leader_election.cc:290] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [CANDIDATE]: Term 1 election: Requested vote from peers a1b8d42ac5424bbe82ef1a3baa7f5f5b (127.16.93.194:36275), 62fbef1b58b946b1977e77b39dfbaf62 (127.16.93.195:46389)
I20250812 19:57:49.118479 18547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d284e91775c84020929b4378f7af93c1" candidate_uuid: "7df4d82dafc64d678150162ecb4cb30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b"
I20250812 19:57:49.119017 18547 raft_consensus.cc:3058] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:49.118912 18680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d284e91775c84020929b4378f7af93c1" candidate_uuid: "7df4d82dafc64d678150162ecb4cb30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fbef1b58b946b1977e77b39dfbaf62"
I20250812 19:57:49.119419 18680 raft_consensus.cc:3058] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Advancing to term 1
W20250812 19:57:49.120875 18460 tablet.cc:2378] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:57:49.125412 18680 raft_consensus.cc:2466] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df4d82dafc64d678150162ecb4cb30f in term 1.
I20250812 19:57:49.126435 18346 leader_election.cc:304] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fbef1b58b946b1977e77b39dfbaf62, 7df4d82dafc64d678150162ecb4cb30f; no voters:
I20250812 19:57:49.127223 18752 raft_consensus.cc:2802] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:49.127743 18414 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d284e91775c84020929b4378f7af93c1" candidate_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4d82dafc64d678150162ecb4cb30f" is_pre_election: true
I20250812 19:57:49.128062 18547 raft_consensus.cc:2466] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df4d82dafc64d678150162ecb4cb30f in term 1.
I20250812 19:57:49.129431 18752 raft_consensus.cc:695] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [term 1 LEADER]: Becoming Leader. State: Replica: 7df4d82dafc64d678150162ecb4cb30f, State: Running, Role: LEADER
I20250812 19:57:49.130126 18680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d284e91775c84020929b4378f7af93c1" candidate_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fbef1b58b946b1977e77b39dfbaf62" is_pre_election: true
I20250812 19:57:49.130316 18752 consensus_queue.cc:237] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } }
I20250812 19:57:49.130708 18680 raft_consensus.cc:2391] T d284e91775c84020929b4378f7af93c1 P 62fbef1b58b946b1977e77b39dfbaf62 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a1b8d42ac5424bbe82ef1a3baa7f5f5b in current term 1: Already voted for candidate 7df4d82dafc64d678150162ecb4cb30f in this term.
I20250812 19:57:49.140942 18482 leader_election.cc:304] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b8d42ac5424bbe82ef1a3baa7f5f5b; no voters: 62fbef1b58b946b1977e77b39dfbaf62, 7df4d82dafc64d678150162ecb4cb30f
I20250812 19:57:49.141803 18751 raft_consensus.cc:2747] T d284e91775c84020929b4378f7af93c1 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250812 19:57:49.142889 18269 catalog_manager.cc:5582] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df4d82dafc64d678150162ecb4cb30f (127.16.93.193). New cstate: current_term: 1 leader_uuid: "7df4d82dafc64d678150162ecb4cb30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } health_report { overall_health: HEALTHY } } }
I20250812 19:57:49.197649 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:57:49.201050 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7df4d82dafc64d678150162ecb4cb30f to finish bootstrapping
I20250812 19:57:49.217711 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a1b8d42ac5424bbe82ef1a3baa7f5f5b to finish bootstrapping
I20250812 19:57:49.228612 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 62fbef1b58b946b1977e77b39dfbaf62 to finish bootstrapping
I20250812 19:57:49.242218 18269 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52554:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250812 19:57:49.243722 18269 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 19:57:49.260406 18394 tablet_service.cc:1468] Processing CreateTablet for tablet f732671cf16e42c2b183cf3f99a5ed03 (DEFAULT_TABLE table=TestAnotherTable [id=f14aabf3351048e99077944fcca91457]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250812 19:57:49.261540 18394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f732671cf16e42c2b183cf3f99a5ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:49.261721 18660 tablet_service.cc:1468] Processing CreateTablet for tablet f732671cf16e42c2b183cf3f99a5ed03 (DEFAULT_TABLE table=TestAnotherTable [id=f14aabf3351048e99077944fcca91457]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250812 19:57:49.262818 18660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f732671cf16e42c2b183cf3f99a5ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:49.262548 18527 tablet_service.cc:1468] Processing CreateTablet for tablet f732671cf16e42c2b183cf3f99a5ed03 (DEFAULT_TABLE table=TestAnotherTable [id=f14aabf3351048e99077944fcca91457]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250812 19:57:49.263612 18527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f732671cf16e42c2b183cf3f99a5ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:57:49.276222 18745 tablet_bootstrap.cc:492] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f: Bootstrap starting.
I20250812 19:57:49.279219 18746 tablet_bootstrap.cc:492] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Bootstrap starting.
I20250812 19:57:49.280717 18744 tablet_bootstrap.cc:492] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62: Bootstrap starting.
I20250812 19:57:49.282759 18745 tablet_bootstrap.cc:654] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:49.285624 18746 tablet_bootstrap.cc:654] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:49.287249 18744 tablet_bootstrap.cc:654] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62: Neither blocks nor log segments found. Creating new log.
I20250812 19:57:49.294953 18745 tablet_bootstrap.cc:492] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f: No bootstrap required, opened a new log
I20250812 19:57:49.295310 18746 tablet_bootstrap.cc:492] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: No bootstrap required, opened a new log
I20250812 19:57:49.295572 18745 ts_tablet_manager.cc:1397] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20250812 19:57:49.295749 18746 ts_tablet_manager.cc:1397] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20250812 19:57:49.296303 18744 tablet_bootstrap.cc:492] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62: No bootstrap required, opened a new log
I20250812 19:57:49.296784 18744 ts_tablet_manager.cc:1397] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20250812 19:57:49.298578 18745 raft_consensus.cc:357] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.298620 18746 raft_consensus.cc:357] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.299600 18746 raft_consensus.cc:383] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:49.299782 18745 raft_consensus.cc:383] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:49.299908 18746 raft_consensus.cc:738] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b8d42ac5424bbe82ef1a3baa7f5f5b, State: Initialized, Role: FOLLOWER
I20250812 19:57:49.299877 18744 raft_consensus.cc:357] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.300246 18745 raft_consensus.cc:738] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df4d82dafc64d678150162ecb4cb30f, State: Initialized, Role: FOLLOWER
I20250812 19:57:49.300570 18744 raft_consensus.cc:383] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:57:49.300894 18744 raft_consensus.cc:738] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fbef1b58b946b1977e77b39dfbaf62, State: Initialized, Role: FOLLOWER
I20250812 19:57:49.300745 18746 consensus_queue.cc:260] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.301146 18745 consensus_queue.cc:260] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.301549 18744 consensus_queue.cc:260] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.303686 18744 ts_tablet_manager.cc:1428] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250812 19:57:49.305816 18746 ts_tablet_manager.cc:1428] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20250812 19:57:49.305954 18745 ts_tablet_manager.cc:1428] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250812 19:57:49.311900 18751 raft_consensus.cc:491] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:57:49.312346 18751 raft_consensus.cc:513] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.314291 18751 leader_election.cc:290] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df4d82dafc64d678150162ecb4cb30f (127.16.93.193:33997), 62fbef1b58b946b1977e77b39dfbaf62 (127.16.93.195:46389)
I20250812 19:57:49.315568 18680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f732671cf16e42c2b183cf3f99a5ed03" candidate_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fbef1b58b946b1977e77b39dfbaf62" is_pre_election: true
I20250812 19:57:49.315836 18414 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f732671cf16e42c2b183cf3f99a5ed03" candidate_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4d82dafc64d678150162ecb4cb30f" is_pre_election: true
I20250812 19:57:49.316154 18680 raft_consensus.cc:2466] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b8d42ac5424bbe82ef1a3baa7f5f5b in term 0.
I20250812 19:57:49.316397 18414 raft_consensus.cc:2466] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b8d42ac5424bbe82ef1a3baa7f5f5b in term 0.
I20250812 19:57:49.317598 18480 leader_election.cc:304] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fbef1b58b946b1977e77b39dfbaf62, a1b8d42ac5424bbe82ef1a3baa7f5f5b; no voters:
I20250812 19:57:49.318598 18751 raft_consensus.cc:2802] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:57:49.318948 18751 raft_consensus.cc:491] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:57:49.319237 18751 raft_consensus.cc:3058] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:49.324011 18751 raft_consensus.cc:513] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.325775 18751 leader_election.cc:290] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [CANDIDATE]: Term 1 election: Requested vote from peers 7df4d82dafc64d678150162ecb4cb30f (127.16.93.193:33997), 62fbef1b58b946b1977e77b39dfbaf62 (127.16.93.195:46389)
I20250812 19:57:49.326303 18414 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f732671cf16e42c2b183cf3f99a5ed03" candidate_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df4d82dafc64d678150162ecb4cb30f"
I20250812 19:57:49.326876 18414 raft_consensus.cc:3058] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:49.327040 18680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f732671cf16e42c2b183cf3f99a5ed03" candidate_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fbef1b58b946b1977e77b39dfbaf62"
I20250812 19:57:49.327581 18680 raft_consensus.cc:3058] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:57:49.333714 18414 raft_consensus.cc:2466] T f732671cf16e42c2b183cf3f99a5ed03 P 7df4d82dafc64d678150162ecb4cb30f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b8d42ac5424bbe82ef1a3baa7f5f5b in term 1.
I20250812 19:57:49.333916 18680 raft_consensus.cc:2466] T f732671cf16e42c2b183cf3f99a5ed03 P 62fbef1b58b946b1977e77b39dfbaf62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b8d42ac5424bbe82ef1a3baa7f5f5b in term 1.
I20250812 19:57:49.334772 18482 leader_election.cc:304] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df4d82dafc64d678150162ecb4cb30f, a1b8d42ac5424bbe82ef1a3baa7f5f5b; no voters:
I20250812 19:57:49.335381 18751 raft_consensus.cc:2802] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:57:49.337101 18751 raft_consensus.cc:695] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [term 1 LEADER]: Becoming Leader. State: Replica: a1b8d42ac5424bbe82ef1a3baa7f5f5b, State: Running, Role: LEADER
I20250812 19:57:49.338089 18751 consensus_queue.cc:237] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } }
I20250812 19:57:49.412422 18270 catalog_manager.cc:5582] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b reported cstate change: term changed from 0 to 1, leader changed from <none> to a1b8d42ac5424bbe82ef1a3baa7f5f5b (127.16.93.194). New cstate: current_term: 1 leader_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 } health_report { overall_health: UNKNOWN } } }
I20250812 19:57:49.546247 18752 consensus_queue.cc:1035] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 19:57:49.564229 18752 consensus_queue.cc:1035] T d284e91775c84020929b4378f7af93c1 P 7df4d82dafc64d678150162ecb4cb30f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b8d42ac5424bbe82ef1a3baa7f5f5b" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250812 19:57:49.772459 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:49.773146 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:49.806070 18765 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250812 19:57:49.879611 18751 consensus_queue.cc:1035] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df4d82dafc64d678150162ecb4cb30f" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250812 19:57:49.936964 18781 consensus_queue.cc:1035] T f732671cf16e42c2b183cf3f99a5ed03 P a1b8d42ac5424bbe82ef1a3baa7f5f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fbef1b58b946b1977e77b39dfbaf62" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250812 19:57:50.391822 18588 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6ec0 after lost signal to thread 18464
W20250812 19:57:50.393307 18588 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0760 after lost signal to thread 18591
W20250812 19:57:51.252389 18775 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 18765
W20250812 19:57:51.253062 18775 kernel_stack_watchdog.cc:198] Thread 18765 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 406ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:57:51.284825 18765 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.441s user 0.567s sys 0.871s
W20250812 19:57:51.403080 18765 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.559s user 0.572s sys 0.880s
W20250812 19:57:52.787771 18793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:52.788395 18793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:52.820842 18793 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 19:57:54.124673 18793 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.262s user 0.000s sys 0.002s
W20250812 19:57:54.124963 18793 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.262s user 0.000s sys 0.005s
W20250812 19:57:55.502960 18811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:55.503579 18811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:55.535218 18811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 19:57:56.821547 18811 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.245s user 0.450s sys 0.794s
W20250812 19:57:56.821883 18811 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.246s user 0.450s sys 0.794s
W20250812 19:57:58.202549 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:57:58.203099 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:57:58.234069 18825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 19:57:59.500620 18825 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.227s user 0.446s sys 0.777s
W20250812 19:57:59.501024 18825 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.228s user 0.450s sys 0.777s
I20250812 19:58:00.593353 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 18329
I20250812 19:58:00.620710 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 18463
I20250812 19:58:00.647735 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 18596
I20250812 19:58:00.672667 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 18237
2025-08-12T19:58:00Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (20992 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250812 19:58:00.732999 16759 test_util.cc:276] Using random seed: 1964372268
I20250812 19:58:00.737118 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:58:00.737334 16759 ts_itest-base.cc:116] --------------
I20250812 19:58:00.737521 16759 ts_itest-base.cc:117] 3 tablet servers
I20250812 19:58:00.737692 16759 ts_itest-base.cc:118] 3 replicas per TS
I20250812 19:58:00.737867 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:58:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:58:00Z Disabled control of system clock
I20250812 19:58:00.773305 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:40025
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:40025
--superuser_acl=no-such-user with env {}
W20250812 19:58:01.072863 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:01.073401 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:01.073853 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:01.105137 18847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:58:01.105420 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:01.105633 18847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:58:01.105827 18847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:58:01.141330 18847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:40025
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:01.142592 18847 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:01.144200 18847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:01.154953 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:02.558540 18852 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 18847
W20250812 19:58:02.956430 18847 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.802s user 0.644s sys 1.156s
W20250812 19:58:02.956995 18847 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.802s user 0.645s sys 1.156s
W20250812 19:58:01.154997 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:02.957993 18855 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1802 milliseconds
I20250812 19:58:02.959136 18847 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 19:58:02.959205 18856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:02.962034 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:02.964373 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:02.965710 18847 hybrid_clock.cc:648] HybridClock initialized: now 1755028682965700 us; error 59 us; skew 500 ppm
I20250812 19:58:02.966513 18847 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:02.972877 18847 webserver.cc:480] Webserver started at http://127.16.93.254:36823/ using document root <none> and password file <none>
I20250812 19:58:02.973803 18847 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:02.973994 18847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:02.974397 18847 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:02.978785 18847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd"
format_stamp: "Formatted at 2025-08-12 19:58:02 on dist-test-slave-wz96"
I20250812 19:58:02.979830 18847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd"
format_stamp: "Formatted at 2025-08-12 19:58:02 on dist-test-slave-wz96"
I20250812 19:58:02.986837 18847 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250812 19:58:02.992249 18863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:02.993342 18847 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250812 19:58:02.993619 18847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd"
format_stamp: "Formatted at 2025-08-12 19:58:02 on dist-test-slave-wz96"
I20250812 19:58:02.993927 18847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:03.051764 18847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:03.053229 18847 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:03.053638 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:03.122480 18847 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:40025
I20250812 19:58:03.122550 18914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:40025 every 8 connection(s)
I20250812 19:58:03.125214 18847 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:58:03.130079 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 18847
I20250812 19:58:03.130410 18915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:03.130479 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:58:03.153424 18915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd: Bootstrap starting.
I20250812 19:58:03.159587 18915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:03.161578 18915 log.cc:826] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:03.166064 18915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd: No bootstrap required, opened a new log
I20250812 19:58:03.183148 18915 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 40025 } }
I20250812 19:58:03.183804 18915 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:03.184016 18915 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9db78f737c4ab28a9231f1c2b0a8cd, State: Initialized, Role: FOLLOWER
I20250812 19:58:03.184692 18915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 40025 } }
I20250812 19:58:03.185166 18915 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:03.185391 18915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:03.185663 18915 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:03.189893 18915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 40025 } }
I20250812 19:58:03.190529 18915 leader_election.cc:304] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b9db78f737c4ab28a9231f1c2b0a8cd; no voters:
I20250812 19:58:03.192063 18915 leader_election.cc:290] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:58:03.192728 18920 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:58:03.194990 18920 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [term 1 LEADER]: Becoming Leader. State: Replica: 6b9db78f737c4ab28a9231f1c2b0a8cd, State: Running, Role: LEADER
I20250812 19:58:03.195760 18920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 40025 } }
I20250812 19:58:03.196730 18915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:58:03.205700 18922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b9db78f737c4ab28a9231f1c2b0a8cd. Latest consensus state: current_term: 1 leader_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 40025 } } }
I20250812 19:58:03.206457 18922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:03.205562 18921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9db78f737c4ab28a9231f1c2b0a8cd" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 40025 } } }
I20250812 19:58:03.208042 18921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:03.212778 18930 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:58:03.222836 18930 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:58:03.237950 18930 catalog_manager.cc:1349] Generated new cluster ID: 7f2b51b22a1c4d06913d593f94fcf169
I20250812 19:58:03.238246 18930 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:58:03.267448 18930 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:58:03.268918 18930 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:58:03.280649 18930 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6b9db78f737c4ab28a9231f1c2b0a8cd: Generated new TSK 0
I20250812 19:58:03.281651 18930 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:58:03.300257 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:40025
--builtin_ntp_servers=127.16.93.212:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 19:58:03.609016 18939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:03.609549 18939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:03.610051 18939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:03.641373 18939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:03.642305 18939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:58:03.691792 18939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:40025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:03.693603 18939 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:03.695892 18939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:03.713706 18945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:05.117751 18944 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 18939
W20250812 19:58:05.495048 18944 kernel_stack_watchdog.cc:198] Thread 18939 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:58:03.717152 18946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:05.496371 18947 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1781 milliseconds
W20250812 19:58:05.496855 18939 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.781s user 0.544s sys 1.059s
W20250812 19:58:05.497207 18939 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.782s user 0.544s sys 1.059s
W20250812 19:58:05.498441 18948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:05.498380 18939 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:05.501433 18939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:05.503640 18939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:05.504987 18939 hybrid_clock.cc:648] HybridClock initialized: now 1755028685504958 us; error 42 us; skew 500 ppm
I20250812 19:58:05.505767 18939 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:05.512419 18939 webserver.cc:480] Webserver started at http://127.16.93.193:41977/ using document root <none> and password file <none>
I20250812 19:58:05.513450 18939 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:05.513643 18939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:05.514111 18939 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:05.518491 18939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a9af7377b30e45f797382a522cfd44a5"
format_stamp: "Formatted at 2025-08-12 19:58:05 on dist-test-slave-wz96"
I20250812 19:58:05.519584 18939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a9af7377b30e45f797382a522cfd44a5"
format_stamp: "Formatted at 2025-08-12 19:58:05 on dist-test-slave-wz96"
I20250812 19:58:05.527071 18939 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250812 19:58:05.533118 18956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:05.534160 18939 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250812 19:58:05.534471 18939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a9af7377b30e45f797382a522cfd44a5"
format_stamp: "Formatted at 2025-08-12 19:58:05 on dist-test-slave-wz96"
I20250812 19:58:05.534829 18939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:05.586895 18939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:05.588359 18939 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:05.588814 18939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:05.591827 18939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:05.596015 18939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:58:05.596221 18939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:05.596499 18939 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:58:05.596697 18939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250812 19:58:05.771049 18939 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:40641
I20250812 19:58:05.771219 19068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:40641 every 8 connection(s)
I20250812 19:58:05.773669 18939 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:58:05.774642 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 18939
I20250812 19:58:05.775252 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:58:05.787060 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:40025
--builtin_ntp_servers=127.16.93.212:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:05.796869 19069 heartbeater.cc:344] Connected to a master server at 127.16.93.254:40025
I20250812 19:58:05.797384 19069 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:05.798648 19069 heartbeater.cc:507] Master 127.16.93.254:40025 requested a full tablet report, sending...
I20250812 19:58:05.801656 18880 ts_manager.cc:194] Registered new tserver with Master: a9af7377b30e45f797382a522cfd44a5 (127.16.93.193:40641)
I20250812 19:58:05.804558 18880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:54391
W20250812 19:58:06.091861 19073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:06.092325 19073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:06.092845 19073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:06.128413 19073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:06.129776 19073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:58:06.171844 19073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:40025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:06.173106 19073 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:06.174594 19073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:06.186038 19079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:06.809074 19069 heartbeater.cc:499] Master 127.16.93.254:40025 was elected leader, sending a full tablet report...
W20250812 19:58:06.186582 19080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:07.590536 19078 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19073
W20250812 19:58:07.827637 19078 kernel_stack_watchdog.cc:198] Thread 19073 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:58:07.833292 19073 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.647s user 0.568s sys 1.078s
W20250812 19:58:07.833724 19073 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.648s user 0.569s sys 1.079s
W20250812 19:58:07.834069 19081 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1646 milliseconds
W20250812 19:58:07.835090 19082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:07.835137 19073 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:07.836239 19073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:07.838253 19073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:07.839591 19073 hybrid_clock.cc:648] HybridClock initialized: now 1755028687839558 us; error 29 us; skew 500 ppm
I20250812 19:58:07.840324 19073 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:07.846154 19073 webserver.cc:480] Webserver started at http://127.16.93.194:38197/ using document root <none> and password file <none>
I20250812 19:58:07.847061 19073 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:07.847273 19073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:07.847723 19073 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:07.852798 19073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "76191cf21b414dd4b340776c1d93a1b7"
format_stamp: "Formatted at 2025-08-12 19:58:07 on dist-test-slave-wz96"
I20250812 19:58:07.854043 19073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "76191cf21b414dd4b340776c1d93a1b7"
format_stamp: "Formatted at 2025-08-12 19:58:07 on dist-test-slave-wz96"
I20250812 19:58:07.860914 19073 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250812 19:58:07.866359 19089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250812 19:58:07.866950 19065 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac060 after lost signal to thread 18940
I20250812 19:58:07.867475 19073 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250812 19:58:07.867882 19073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "76191cf21b414dd4b340776c1d93a1b7"
format_stamp: "Formatted at 2025-08-12 19:58:07 on dist-test-slave-wz96"
W20250812 19:58:07.868094 19065 debug-util.cc:398] Leaking SignalData structure 0x7b08000acf40 after lost signal to thread 19068
I20250812 19:58:07.868311 19073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:07.940888 19073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:07.942291 19073 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:07.942708 19073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:07.945286 19073 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:07.949813 19073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:58:07.950016 19073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:07.950270 19073 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:58:07.950433 19073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:08.121330 19073 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:32799
I20250812 19:58:08.121510 19201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:32799 every 8 connection(s)
I20250812 19:58:08.124917 19073 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:58:08.128255 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 19073
I20250812 19:58:08.128775 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 19:58:08.137194 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:40025
--builtin_ntp_servers=127.16.93.212:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:08.185853 19202 heartbeater.cc:344] Connected to a master server at 127.16.93.254:40025
I20250812 19:58:08.186352 19202 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:08.187702 19202 heartbeater.cc:507] Master 127.16.93.254:40025 requested a full tablet report, sending...
I20250812 19:58:08.190423 18880 ts_manager.cc:194] Registered new tserver with Master: 76191cf21b414dd4b340776c1d93a1b7 (127.16.93.194:32799)
I20250812 19:58:08.192252 18880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:46311
W20250812 19:58:08.460000 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:08.460546 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:08.461102 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:08.492744 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:08.493594 19204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:58:08.530001 19204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:40025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:08.531325 19204 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:08.532927 19204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:08.546537 19212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:09.196072 19202 heartbeater.cc:499] Master 127.16.93.254:40025 was elected leader, sending a full tablet report...
W20250812 19:58:09.949308 19211 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19204
W20250812 19:58:10.280213 19211 kernel_stack_watchdog.cc:198] Thread 19204 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:58:08.548198 19213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:10.280895 19214 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1734 milliseconds
W20250812 19:58:10.281178 19204 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.735s user 0.483s sys 1.102s
W20250812 19:58:10.281567 19204 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.736s user 0.483s sys 1.102s
I20250812 19:58:10.282048 19204 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 19:58:10.282065 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:10.286628 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:10.289249 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:10.290735 19204 hybrid_clock.cc:648] HybridClock initialized: now 1755028690290679 us; error 50 us; skew 500 ppm
I20250812 19:58:10.291841 19204 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:10.299614 19204 webserver.cc:480] Webserver started at http://127.16.93.195:44269/ using document root <none> and password file <none>
I20250812 19:58:10.300936 19204 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:10.301231 19204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:10.301795 19204 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:10.308512 19204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3d5e490614004bbb9fff951377adcdaf"
format_stamp: "Formatted at 2025-08-12 19:58:10 on dist-test-slave-wz96"
I20250812 19:58:10.310357 19204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3d5e490614004bbb9fff951377adcdaf"
format_stamp: "Formatted at 2025-08-12 19:58:10 on dist-test-slave-wz96"
I20250812 19:58:10.318866 19204 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20250812 19:58:10.324988 19222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:10.326346 19204 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250812 19:58:10.326782 19204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3d5e490614004bbb9fff951377adcdaf"
format_stamp: "Formatted at 2025-08-12 19:58:10 on dist-test-slave-wz96"
I20250812 19:58:10.327268 19204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:10.403077 19204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:10.404978 19204 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:10.405427 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:10.408087 19204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:10.412432 19204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:58:10.412678 19204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250812 19:58:10.412914 19204 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:58:10.413081 19204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:10.546464 19204 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:45639
I20250812 19:58:10.546566 19334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:45639 every 8 connection(s)
I20250812 19:58:10.549043 19204 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:58:10.551066 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 19204
I20250812 19:58:10.551599 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 19:58:10.569769 19335 heartbeater.cc:344] Connected to a master server at 127.16.93.254:40025
I20250812 19:58:10.570233 19335 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:10.571406 19335 heartbeater.cc:507] Master 127.16.93.254:40025 requested a full tablet report, sending...
I20250812 19:58:10.573650 18880 ts_manager.cc:194] Registered new tserver with Master: 3d5e490614004bbb9fff951377adcdaf (127.16.93.195:45639)
I20250812 19:58:10.574865 18880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:36735
I20250812 19:58:10.586602 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:10.619693 18880 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46564:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250812 19:58:10.638051 18880 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 19:58:10.687429 19137 tablet_service.cc:1468] Processing CreateTablet for tablet d9b849f26f7f4e45b5bbd21acae970dc (DEFAULT_TABLE table=TestTable [id=82c7210e6d5e44c98d520ae6ae195fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:58:10.689343 19137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b849f26f7f4e45b5bbd21acae970dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:10.689853 19004 tablet_service.cc:1468] Processing CreateTablet for tablet d9b849f26f7f4e45b5bbd21acae970dc (DEFAULT_TABLE table=TestTable [id=82c7210e6d5e44c98d520ae6ae195fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:58:10.691687 19004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b849f26f7f4e45b5bbd21acae970dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:10.695160 19270 tablet_service.cc:1468] Processing CreateTablet for tablet d9b849f26f7f4e45b5bbd21acae970dc (DEFAULT_TABLE table=TestTable [id=82c7210e6d5e44c98d520ae6ae195fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:58:10.696959 19270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b849f26f7f4e45b5bbd21acae970dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:10.719260 19355 tablet_bootstrap.cc:492] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: Bootstrap starting.
I20250812 19:58:10.722328 19356 tablet_bootstrap.cc:492] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: Bootstrap starting.
I20250812 19:58:10.722618 19354 tablet_bootstrap.cc:492] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: Bootstrap starting.
I20250812 19:58:10.727476 19355 tablet_bootstrap.cc:654] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:10.729189 19356 tablet_bootstrap.cc:654] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:10.729974 19355 log.cc:826] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:10.730472 19354 tablet_bootstrap.cc:654] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:10.731652 19356 log.cc:826] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:10.732990 19354 log.cc:826] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:10.736119 19355 tablet_bootstrap.cc:492] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: No bootstrap required, opened a new log
I20250812 19:58:10.736661 19355 ts_tablet_manager.cc:1397] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.006s
I20250812 19:58:10.738111 19354 tablet_bootstrap.cc:492] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: No bootstrap required, opened a new log
I20250812 19:58:10.738503 19354 ts_tablet_manager.cc:1397] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250812 19:58:10.739094 19356 tablet_bootstrap.cc:492] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: No bootstrap required, opened a new log
I20250812 19:58:10.739570 19356 ts_tablet_manager.cc:1397] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20250812 19:58:10.756181 19354 raft_consensus.cc:357] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.757177 19354 raft_consensus.cc:383] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:10.757496 19354 raft_consensus.cc:738] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76191cf21b414dd4b340776c1d93a1b7, State: Initialized, Role: FOLLOWER
I20250812 19:58:10.758503 19354 consensus_queue.cc:260] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.762776 19354 ts_tablet_manager.cc:1428] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: Time spent starting tablet: real 0.024s user 0.025s sys 0.000s
I20250812 19:58:10.763295 19355 raft_consensus.cc:357] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.764230 19355 raft_consensus.cc:383] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:10.764525 19355 raft_consensus.cc:738] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5e490614004bbb9fff951377adcdaf, State: Initialized, Role: FOLLOWER
I20250812 19:58:10.765437 19355 consensus_queue.cc:260] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.766137 19356 raft_consensus.cc:357] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.767179 19356 raft_consensus.cc:383] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:10.767583 19356 raft_consensus.cc:738] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9af7377b30e45f797382a522cfd44a5, State: Initialized, Role: FOLLOWER
I20250812 19:58:10.768762 19335 heartbeater.cc:499] Master 127.16.93.254:40025 was elected leader, sending a full tablet report...
I20250812 19:58:10.768682 19356 consensus_queue.cc:260] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.769767 19355 ts_tablet_manager.cc:1428] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: Time spent starting tablet: real 0.033s user 0.032s sys 0.000s
I20250812 19:58:10.774654 19356 ts_tablet_manager.cc:1428] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: Time spent starting tablet: real 0.035s user 0.036s sys 0.000s
W20250812 19:58:10.798127 19070 tablet.cc:2378] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 19:58:10.803424 19336 tablet.cc:2378] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:58:10.809049 19362 raft_consensus.cc:491] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:58:10.809509 19362 raft_consensus.cc:513] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.812043 19362 leader_election.cc:290] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76191cf21b414dd4b340776c1d93a1b7 (127.16.93.194:32799), 3d5e490614004bbb9fff951377adcdaf (127.16.93.195:45639)
I20250812 19:58:10.823915 19290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9b849f26f7f4e45b5bbd21acae970dc" candidate_uuid: "a9af7377b30e45f797382a522cfd44a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5e490614004bbb9fff951377adcdaf" is_pre_election: true
I20250812 19:58:10.823925 19157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9b849f26f7f4e45b5bbd21acae970dc" candidate_uuid: "a9af7377b30e45f797382a522cfd44a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76191cf21b414dd4b340776c1d93a1b7" is_pre_election: true
I20250812 19:58:10.824635 19157 raft_consensus.cc:2466] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9af7377b30e45f797382a522cfd44a5 in term 0.
I20250812 19:58:10.824651 19290 raft_consensus.cc:2466] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9af7377b30e45f797382a522cfd44a5 in term 0.
I20250812 19:58:10.825778 18958 leader_election.cc:304] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5e490614004bbb9fff951377adcdaf, a9af7377b30e45f797382a522cfd44a5; no voters:
I20250812 19:58:10.826490 19362 raft_consensus.cc:2802] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:58:10.826836 19362 raft_consensus.cc:491] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:58:10.827087 19362 raft_consensus.cc:3058] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:10.831425 19362 raft_consensus.cc:513] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.832849 19362 leader_election.cc:290] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [CANDIDATE]: Term 1 election: Requested vote from peers 76191cf21b414dd4b340776c1d93a1b7 (127.16.93.194:32799), 3d5e490614004bbb9fff951377adcdaf (127.16.93.195:45639)
I20250812 19:58:10.833595 19157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9b849f26f7f4e45b5bbd21acae970dc" candidate_uuid: "a9af7377b30e45f797382a522cfd44a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76191cf21b414dd4b340776c1d93a1b7"
I20250812 19:58:10.833791 19290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9b849f26f7f4e45b5bbd21acae970dc" candidate_uuid: "a9af7377b30e45f797382a522cfd44a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5e490614004bbb9fff951377adcdaf"
I20250812 19:58:10.834065 19157 raft_consensus.cc:3058] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:10.834259 19290 raft_consensus.cc:3058] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:10.838632 19157 raft_consensus.cc:2466] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9af7377b30e45f797382a522cfd44a5 in term 1.
I20250812 19:58:10.838881 19290 raft_consensus.cc:2466] T d9b849f26f7f4e45b5bbd21acae970dc P 3d5e490614004bbb9fff951377adcdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9af7377b30e45f797382a522cfd44a5 in term 1.
I20250812 19:58:10.839643 18960 leader_election.cc:304] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76191cf21b414dd4b340776c1d93a1b7, a9af7377b30e45f797382a522cfd44a5; no voters:
I20250812 19:58:10.840283 19362 raft_consensus.cc:2802] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:58:10.841771 19362 raft_consensus.cc:695] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [term 1 LEADER]: Becoming Leader. State: Replica: a9af7377b30e45f797382a522cfd44a5, State: Running, Role: LEADER
I20250812 19:58:10.842474 19362 consensus_queue.cc:237] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } }
I20250812 19:58:10.851473 18879 catalog_manager.cc:5582] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9af7377b30e45f797382a522cfd44a5 (127.16.93.193). New cstate: current_term: 1 leader_uuid: "a9af7377b30e45f797382a522cfd44a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9af7377b30e45f797382a522cfd44a5" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 40641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 } health_report { overall_health: UNKNOWN } } }
W20250812 19:58:10.898574 19203 tablet.cc:2378] T d9b849f26f7f4e45b5bbd21acae970dc P 76191cf21b414dd4b340776c1d93a1b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:58:10.899821 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:10.903048 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a9af7377b30e45f797382a522cfd44a5 to finish bootstrapping
I20250812 19:58:10.915145 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 76191cf21b414dd4b340776c1d93a1b7 to finish bootstrapping
I20250812 19:58:10.926085 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3d5e490614004bbb9fff951377adcdaf to finish bootstrapping
I20250812 19:58:11.278739 19362 consensus_queue.cc:1035] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5e490614004bbb9fff951377adcdaf" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 19:58:11.302780 19365 consensus_queue.cc:1035] T d9b849f26f7f4e45b5bbd21acae970dc P a9af7377b30e45f797382a522cfd44a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76191cf21b414dd4b340776c1d93a1b7" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250812 19:58:12.631107 18879 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:46578
I20250812 19:58:13.660447 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 18939
I20250812 19:58:13.685014 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 19073
I20250812 19:58:13.710103 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 19204
I20250812 19:58:13.737040 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 18847
2025-08-12T19:58:13Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (13062 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250812 19:58:13.795890 16759 test_util.cc:276] Using random seed: 1977435135
I20250812 19:58:13.799785 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:58:13.799966 16759 ts_itest-base.cc:116] --------------
I20250812 19:58:13.800088 16759 ts_itest-base.cc:117] 3 tablet servers
I20250812 19:58:13.800199 16759 ts_itest-base.cc:118] 3 replicas per TS
I20250812 19:58:13.800304 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:58:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:58:13Z Disabled control of system clock
I20250812 19:58:13.838188 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35295
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:35295 with env {}
W20250812 19:58:14.133378 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:14.133991 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:14.134459 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:14.165576 19407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:58:14.165881 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:14.166172 19407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:58:14.166411 19407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:58:14.202282 19407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:35295
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:14.203649 19407 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:14.205350 19407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:14.215526 19413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:14.217067 19414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:14.220695 19416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:15.394851 19415 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 19:58:15.394901 19407 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:15.399173 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:15.401810 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:15.403151 19407 hybrid_clock.cc:648] HybridClock initialized: now 1755028695403122 us; error 64 us; skew 500 ppm
I20250812 19:58:15.404014 19407 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:15.410020 19407 webserver.cc:480] Webserver started at http://127.16.93.254:45775/ using document root <none> and password file <none>
I20250812 19:58:15.410923 19407 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:15.411129 19407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:15.411578 19407 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:15.416381 19407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fd532d514cb443c18faf85b14f1a5eb7"
format_stamp: "Formatted at 2025-08-12 19:58:15 on dist-test-slave-wz96"
I20250812 19:58:15.417510 19407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fd532d514cb443c18faf85b14f1a5eb7"
format_stamp: "Formatted at 2025-08-12 19:58:15 on dist-test-slave-wz96"
I20250812 19:58:15.425060 19407 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250812 19:58:15.430495 19423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:15.431437 19407 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250812 19:58:15.431736 19407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fd532d514cb443c18faf85b14f1a5eb7"
format_stamp: "Formatted at 2025-08-12 19:58:15 on dist-test-slave-wz96"
I20250812 19:58:15.432039 19407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:15.502647 19407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:15.504158 19407 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:15.504616 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:15.573666 19407 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:35295
I20250812 19:58:15.573738 19474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:35295 every 8 connection(s)
I20250812 19:58:15.576282 19407 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:58:15.577083 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 19407
I20250812 19:58:15.577641 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:58:15.582432 19475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:15.606302 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap starting.
I20250812 19:58:15.611852 19475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:15.613550 19475 log.cc:826] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:15.617763 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: No bootstrap required, opened a new log
I20250812 19:58:15.635146 19475 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:15.635802 19475 raft_consensus.cc:383] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:15.636045 19475 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Initialized, Role: FOLLOWER
I20250812 19:58:15.636720 19475 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:15.637271 19475 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:15.637605 19475 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:15.637971 19475 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:15.641940 19475 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:15.642603 19475 leader_election.cc:304] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd532d514cb443c18faf85b14f1a5eb7; no voters:
I20250812 19:58:15.644207 19475 leader_election.cc:290] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:58:15.644894 19480 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:58:15.647032 19480 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 1 LEADER]: Becoming Leader. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Running, Role: LEADER
I20250812 19:58:15.647763 19480 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:15.648903 19475 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:58:15.658922 19481 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:15.659093 19482 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd532d514cb443c18faf85b14f1a5eb7. Latest consensus state: current_term: 1 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:15.659833 19481 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:15.659883 19482 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:15.665129 19490 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:58:15.677917 19490 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:58:15.693367 19490 catalog_manager.cc:1349] Generated new cluster ID: bb833e26fffb4fe9bbcca16f5584a760
I20250812 19:58:15.693694 19490 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:58:15.723423 19490 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:58:15.725417 19490 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:58:15.739617 19490 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Generated new TSK 0
I20250812 19:58:15.740703 19490 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:58:15.750042 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35295
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 19:58:16.048223 19499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:16.048775 19499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:16.049319 19499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:16.082192 19499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:16.083068 19499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:58:16.118468 19499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:16.119792 19499 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:16.121398 19499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:16.133978 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:17.537217 19504 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19499
W20250812 19:58:17.589732 19504 kernel_stack_watchdog.cc:198] Thread 19499 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:58:16.135207 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:17.596264 19507 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1459 milliseconds
W20250812 19:58:17.597497 19499 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.462s user 0.000s sys 0.006s
W20250812 19:58:17.597800 19499 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.462s user 0.000s sys 0.006s
W20250812 19:58:17.599275 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:17.599337 19499 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:17.600561 19499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:17.603153 19499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:17.604672 19499 hybrid_clock.cc:648] HybridClock initialized: now 1755028697604636 us; error 46 us; skew 500 ppm
I20250812 19:58:17.605458 19499 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:17.612480 19499 webserver.cc:480] Webserver started at http://127.16.93.193:41145/ using document root <none> and password file <none>
I20250812 19:58:17.613600 19499 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:17.613870 19499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:17.614326 19499 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:17.618821 19499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc"
format_stamp: "Formatted at 2025-08-12 19:58:17 on dist-test-slave-wz96"
I20250812 19:58:17.619957 19499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc"
format_stamp: "Formatted at 2025-08-12 19:58:17 on dist-test-slave-wz96"
I20250812 19:58:17.627614 19499 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250812 19:58:17.634073 19515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:17.635174 19499 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 19:58:17.635497 19499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc"
format_stamp: "Formatted at 2025-08-12 19:58:17 on dist-test-slave-wz96"
I20250812 19:58:17.635807 19499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:17.688746 19499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:17.690248 19499 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:17.690680 19499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:17.693724 19499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:17.698056 19499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:58:17.698246 19499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:17.698510 19499 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:58:17.698673 19499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:17.868039 19499 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:41115
I20250812 19:58:17.868206 19627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:41115 every 8 connection(s)
I20250812 19:58:17.870597 19499 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:58:17.871249 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 19499
I20250812 19:58:17.871713 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:58:17.880687 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35295
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:17.911078 19628 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:17.911779 19628 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:17.913414 19628 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:17.916785 19440 ts_manager.cc:194] Registered new tserver with Master: cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193:41115)
I20250812 19:58:17.919482 19440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:55757
W20250812 19:58:18.203452 19632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:18.203969 19632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:18.204468 19632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:18.237126 19632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:18.237987 19632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:58:18.273310 19632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:18.274649 19632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:18.276185 19632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:18.287281 19638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:18.923900 19628 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
W20250812 19:58:18.288836 19639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:18.291924 19641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:19.443039 19640 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 19:58:19.443781 19632 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:19.447700 19632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:19.449941 19632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:19.451280 19632 hybrid_clock.cc:648] HybridClock initialized: now 1755028699451221 us; error 68 us; skew 500 ppm
I20250812 19:58:19.452054 19632 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:19.458544 19632 webserver.cc:480] Webserver started at http://127.16.93.194:38701/ using document root <none> and password file <none>
I20250812 19:58:19.459439 19632 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:19.459652 19632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:19.460085 19632 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:19.464489 19632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "14f940533a744ceba9ebf397a1a45b8f"
format_stamp: "Formatted at 2025-08-12 19:58:19 on dist-test-slave-wz96"
I20250812 19:58:19.465596 19632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "14f940533a744ceba9ebf397a1a45b8f"
format_stamp: "Formatted at 2025-08-12 19:58:19 on dist-test-slave-wz96"
I20250812 19:58:19.472676 19632 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250812 19:58:19.478459 19648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:19.479641 19632 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250812 19:58:19.480059 19632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14f940533a744ceba9ebf397a1a45b8f"
format_stamp: "Formatted at 2025-08-12 19:58:19 on dist-test-slave-wz96"
I20250812 19:58:19.480532 19632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:19.542001 19632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:19.543463 19632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:19.543892 19632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:19.546345 19632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:19.551028 19632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:58:19.551235 19632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:19.551486 19632 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:58:19.551645 19632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:19.686208 19632 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:41739
I20250812 19:58:19.686324 19760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:41739 every 8 connection(s)
I20250812 19:58:19.688722 19632 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:58:19.692806 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 19632
I20250812 19:58:19.693305 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 19:58:19.700013 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35295
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:19.714354 19761 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:19.714774 19761 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:19.715749 19761 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:19.718070 19440 ts_manager.cc:194] Registered new tserver with Master: 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
I20250812 19:58:19.720060 19440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:51811
W20250812 19:58:20.007911 19765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:20.008389 19765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:20.008846 19765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:20.041121 19765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:20.041954 19765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:58:20.076354 19765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:20.077754 19765 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:20.079352 19765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:20.091076 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:20.723337 19761 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
W20250812 19:58:21.496508 19770 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19765
W20250812 19:58:21.752113 19765 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.661s user 0.614s sys 1.044s
W20250812 19:58:21.752718 19765 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.661s user 0.614s sys 1.044s
W20250812 19:58:20.092212 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:21.754487 19773 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1662 milliseconds
W20250812 19:58:21.755339 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:21.755270 19765 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:21.758432 19765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:21.760411 19765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:21.761758 19765 hybrid_clock.cc:648] HybridClock initialized: now 1755028701761742 us; error 41 us; skew 500 ppm
I20250812 19:58:21.762491 19765 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:21.768275 19765 webserver.cc:480] Webserver started at http://127.16.93.195:40469/ using document root <none> and password file <none>
I20250812 19:58:21.769178 19765 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:21.769392 19765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:21.769798 19765 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:58:21.774168 19765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fb685ac193094374b6df35b06da361e2"
format_stamp: "Formatted at 2025-08-12 19:58:21 on dist-test-slave-wz96"
I20250812 19:58:21.775202 19765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fb685ac193094374b6df35b06da361e2"
format_stamp: "Formatted at 2025-08-12 19:58:21 on dist-test-slave-wz96"
I20250812 19:58:21.782126 19765 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250812 19:58:21.787607 19781 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:21.788552 19765 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250812 19:58:21.788893 19765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb685ac193094374b6df35b06da361e2"
format_stamp: "Formatted at 2025-08-12 19:58:21 on dist-test-slave-wz96"
I20250812 19:58:21.789239 19765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:21.834867 19765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:21.836273 19765 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:21.836719 19765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:21.839138 19765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:21.843104 19765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:58:21.843317 19765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:21.843550 19765 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:58:21.843714 19765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:21.977272 19765 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:45977
I20250812 19:58:21.977363 19893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:45977 every 8 connection(s)
I20250812 19:58:21.979741 19765 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:58:21.980916 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 19765
I20250812 19:58:21.981487 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 19:58:22.000280 19894 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:22.000741 19894 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:22.001888 19894 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:22.003726 19440 ts_manager.cc:194] Registered new tserver with Master: fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:22.004912 19440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:44805
I20250812 19:58:22.015825 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:22.049680 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:22.050011 16759 test_util.cc:276] Using random seed: 1985689289
I20250812 19:58:22.089054 19440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34762:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 19:58:22.128243 19696 tablet_service.cc:1468] Processing CreateTablet for tablet 21fd193e9c454edc98941ed66ec26d5a (DEFAULT_TABLE table=TestTable [id=dbc4e1080ce64f0d926784f4b17d665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:58:22.129628 19696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21fd193e9c454edc98941ed66ec26d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:22.148425 19914 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap starting.
I20250812 19:58:22.154028 19914 tablet_bootstrap.cc:654] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:22.155758 19914 log.cc:826] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:22.160316 19914 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: No bootstrap required, opened a new log
I20250812 19:58:22.160709 19914 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20250812 19:58:22.177623 19914 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:22.178191 19914 raft_consensus.cc:383] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:22.178433 19914 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Initialized, Role: FOLLOWER
I20250812 19:58:22.179098 19914 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:22.179553 19914 raft_consensus.cc:397] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:22.179796 19914 raft_consensus.cc:491] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:22.180079 19914 raft_consensus.cc:3058] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:22.184120 19914 raft_consensus.cc:513] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:22.184922 19914 leader_election.cc:304] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f940533a744ceba9ebf397a1a45b8f; no voters:
I20250812 19:58:22.186537 19914 leader_election.cc:290] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:58:22.186897 19916 raft_consensus.cc:2802] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:58:22.188911 19916 raft_consensus.cc:695] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 LEADER]: Becoming Leader. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Running, Role: LEADER
I20250812 19:58:22.189818 19916 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:22.190387 19914 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Time spent starting tablet: real 0.029s user 0.026s sys 0.002s
W20250812 19:58:22.196214 19762 tablet.cc:2378] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:58:22.203423 19439 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194). New cstate: current_term: 1 leader_uuid: "14f940533a744ceba9ebf397a1a45b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } health_report { overall_health: HEALTHY } } }
I20250812 19:58:22.405005 16759 test_util.cc:276] Using random seed: 1986044276
I20250812 19:58:22.430425 19440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34776:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 19:58:22.457748 19829 tablet_service.cc:1468] Processing CreateTablet for tablet c547a53cb7c647ad8ed0762e9470b5ee (DEFAULT_TABLE table=TestTable1 [id=cf63513a164049e7975825d40a114231]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:58:22.459187 19829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c547a53cb7c647ad8ed0762e9470b5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:22.478964 19934 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap starting.
I20250812 19:58:22.484887 19934 tablet_bootstrap.cc:654] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:22.486585 19934 log.cc:826] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:22.490963 19934 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: No bootstrap required, opened a new log
I20250812 19:58:22.491425 19934 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250812 19:58:22.508962 19934 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:22.509518 19934 raft_consensus.cc:383] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:22.509708 19934 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb685ac193094374b6df35b06da361e2, State: Initialized, Role: FOLLOWER
I20250812 19:58:22.510303 19934 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:22.510764 19934 raft_consensus.cc:397] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:22.510989 19934 raft_consensus.cc:491] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:22.511271 19934 raft_consensus.cc:3058] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:22.515379 19934 raft_consensus.cc:513] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:22.516023 19934 leader_election.cc:304] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb685ac193094374b6df35b06da361e2; no voters:
I20250812 19:58:22.517830 19934 leader_election.cc:290] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:58:22.518280 19936 raft_consensus.cc:2802] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:58:22.520929 19936 raft_consensus.cc:695] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 LEADER]: Becoming Leader. State: Replica: fb685ac193094374b6df35b06da361e2, State: Running, Role: LEADER
I20250812 19:58:22.521490 19894 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
I20250812 19:58:22.522012 19936 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:22.522910 19934 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Time spent starting tablet: real 0.031s user 0.023s sys 0.009s
I20250812 19:58:22.533322 19440 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb685ac193094374b6df35b06da361e2 (127.16.93.195). New cstate: current_term: 1 leader_uuid: "fb685ac193094374b6df35b06da361e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: HEALTHY } } }
I20250812 19:58:22.758489 16759 test_util.cc:276] Using random seed: 1986397757
I20250812 19:58:22.780916 19434 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 19:58:22.808559 19563 tablet_service.cc:1468] Processing CreateTablet for tablet 0b297f6787f94b618af8d9c680ffd066 (DEFAULT_TABLE table=TestTable2 [id=4958118b9eea4eb8a7ad1badb5e9dd61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:58:22.809998 19563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b297f6787f94b618af8d9c680ffd066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:22.829790 19954 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:22.837289 19954 tablet_bootstrap.cc:654] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Neither blocks nor log segments found. Creating new log.
I20250812 19:58:22.839473 19954 log.cc:826] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:22.844519 19954 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: No bootstrap required, opened a new log
I20250812 19:58:22.844971 19954 ts_tablet_manager.cc:1397] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20250812 19:58:22.862529 19954 raft_consensus.cc:357] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:22.863132 19954 raft_consensus.cc:383] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:58:22.863348 19954 raft_consensus.cc:738] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: FOLLOWER
I20250812 19:58:22.864035 19954 consensus_queue.cc:260] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:22.864562 19954 raft_consensus.cc:397] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:22.864873 19954 raft_consensus.cc:491] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:22.865201 19954 raft_consensus.cc:3058] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:58:22.870043 19954 raft_consensus.cc:513] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:22.870806 19954 leader_election.cc:304] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf17e5fbd8e148bfa3aa99f74803a8dc; no voters:
I20250812 19:58:22.872603 19954 leader_election.cc:290] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:58:22.873059 19956 raft_consensus.cc:2802] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:58:22.877228 19954 ts_tablet_manager.cc:1428] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.032s user 0.019s sys 0.012s
I20250812 19:58:22.881098 19956 raft_consensus.cc:695] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 LEADER]: Becoming Leader. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Running, Role: LEADER
I20250812 19:58:22.881911 19956 consensus_queue.cc:237] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:22.895057 19434 catalog_manager.cc:5582] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: term changed from 0 to 1, leader changed from <none> to cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193). New cstate: current_term: 1 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } health_report { overall_health: HEALTHY } } }
I20250812 19:58:23.102986 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 19407
W20250812 19:58:23.218767 19761 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:35295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:35295: connect: Connection refused (error 111)
W20250812 19:58:23.547587 19894 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:35295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:35295: connect: Connection refused (error 111)
W20250812 19:58:23.927191 19628 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:35295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:35295: connect: Connection refused (error 111)
I20250812 19:58:28.240499 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 19499
I20250812 19:58:28.263533 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 19632
I20250812 19:58:28.287732 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 19765
I20250812 19:58:28.314426 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35295
--webserver_interface=127.16.93.254
--webserver_port=45775
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:35295 with env {}
W20250812 19:58:28.614454 20036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:28.615084 20036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:28.615569 20036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:28.646332 20036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:58:28.646649 20036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:28.646936 20036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:58:28.647194 20036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:58:28.681597 20036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:35295
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=45775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:28.682952 20036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:28.684549 20036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:28.695385 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:28.695851 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:28.700289 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:29.880698 20044 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 19:58:29.880828 20036 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:29.884716 20036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:29.887813 20036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:29.889241 20036 hybrid_clock.cc:648] HybridClock initialized: now 1755028709889178 us; error 84 us; skew 500 ppm
I20250812 19:58:29.890079 20036 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:29.897029 20036 webserver.cc:480] Webserver started at http://127.16.93.254:45775/ using document root <none> and password file <none>
I20250812 19:58:29.898054 20036 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:29.898244 20036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:29.905929 20036 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250812 19:58:29.910262 20052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:29.911327 20036 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 19:58:29.911664 20036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fd532d514cb443c18faf85b14f1a5eb7"
format_stamp: "Formatted at 2025-08-12 19:58:15 on dist-test-slave-wz96"
I20250812 19:58:29.913599 20036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:29.964349 20036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:29.965849 20036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:29.966250 20036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:30.037243 20036 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:35295
I20250812 19:58:30.037309 20103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:35295 every 8 connection(s)
I20250812 19:58:30.040004 20036 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:58:30.044420 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20036
I20250812 19:58:30.045851 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:41115
--local_ip_for_outbound_sockets=127.16.93.193
--tserver_master_addrs=127.16.93.254:35295
--webserver_port=41145
--webserver_interface=127.16.93.193
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:30.050354 20104 sys_catalog.cc:263] Verifying existing consensus state
I20250812 19:58:30.054948 20104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap starting.
I20250812 19:58:30.065533 20104 log.cc:826] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:30.111969 20104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:30.112838 20104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap complete.
I20250812 19:58:30.133194 20104 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:30.135288 20104 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Initialized, Role: FOLLOWER
I20250812 19:58:30.136085 20104 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:30.136636 20104 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:30.136909 20104 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:30.137217 20104 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:58:30.142982 20104 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:30.143586 20104 leader_election.cc:304] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd532d514cb443c18faf85b14f1a5eb7; no voters:
I20250812 19:58:30.145680 20104 leader_election.cc:290] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250812 19:58:30.146075 20108 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Leader election won for term 3
I20250812 19:58:30.149205 20108 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 LEADER]: Becoming Leader. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Running, Role: LEADER
I20250812 19:58:30.150012 20108 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:30.150605 20104 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:58:30.161195 20109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:30.161814 20109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:30.160283 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd532d514cb443c18faf85b14f1a5eb7. Latest consensus state: current_term: 3 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:30.164130 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:30.176777 20115 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:58:30.189410 20115 catalog_manager.cc:671] Loaded metadata for table TestTable [id=04bd440bf2214297934d556fc36b02b0]
I20250812 19:58:30.191658 20115 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=4958118b9eea4eb8a7ad1badb5e9dd61]
I20250812 19:58:30.193377 20115 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=f312994f95cf43768326d77ca3a13033]
I20250812 19:58:30.201604 20115 tablet_loader.cc:96] loaded metadata for tablet 0b297f6787f94b618af8d9c680ffd066 (table TestTable2 [id=4958118b9eea4eb8a7ad1badb5e9dd61])
I20250812 19:58:30.202940 20115 tablet_loader.cc:96] loaded metadata for tablet 21fd193e9c454edc98941ed66ec26d5a (table TestTable [id=04bd440bf2214297934d556fc36b02b0])
I20250812 19:58:30.204488 20115 tablet_loader.cc:96] loaded metadata for tablet c547a53cb7c647ad8ed0762e9470b5ee (table TestTable1 [id=f312994f95cf43768326d77ca3a13033])
I20250812 19:58:30.205962 20115 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:58:30.211131 20115 catalog_manager.cc:1261] Loaded cluster ID: bb833e26fffb4fe9bbcca16f5584a760
I20250812 19:58:30.211556 20115 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:58:30.220556 20115 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:58:30.226439 20115 catalog_manager.cc:5966] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Loaded TSK: 0
I20250812 19:58:30.228298 20115 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 19:58:30.381398 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:30.381920 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:30.382442 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:30.413290 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:30.414147 20106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:58:30.449210 20106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:41115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=41145
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:30.450559 20106 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:30.452101 20106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:30.464726 20131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:30.465794 20132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:31.932631 20106 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.467s user 0.526s sys 0.930s
W20250812 19:58:31.933131 20106 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.467s user 0.527s sys 0.930s
W20250812 19:58:31.867455 20130 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 20106
W20250812 19:58:31.934906 20134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:31.938275 20133 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1468 milliseconds
I20250812 19:58:31.938349 20106 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:31.939535 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:31.942039 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:31.943524 20106 hybrid_clock.cc:648] HybridClock initialized: now 1755028711943471 us; error 55 us; skew 500 ppm
I20250812 19:58:31.944298 20106 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:31.950819 20106 webserver.cc:480] Webserver started at http://127.16.93.193:41145/ using document root <none> and password file <none>
I20250812 19:58:31.951727 20106 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:31.951969 20106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:31.959832 20106 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.006s
I20250812 19:58:31.964819 20141 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:31.965870 20106 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250812 19:58:31.966168 20106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc"
format_stamp: "Formatted at 2025-08-12 19:58:17 on dist-test-slave-wz96"
I20250812 19:58:31.967947 20106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:32.029243 20106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:32.030869 20106 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:32.031302 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:32.034298 20106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:32.041060 20148 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 19:58:32.048744 20106 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 19:58:32.048976 20106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.002s
I20250812 19:58:32.049281 20106 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 19:58:32.053859 20106 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 19:58:32.054102 20106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250812 19:58:32.054466 20148 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:32.127390 20148 log.cc:826] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:32.245517 20106 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:41115
I20250812 19:58:32.245752 20255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:41115 every 8 connection(s)
I20250812 19:58:32.249145 20106 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:58:32.250339 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20106
I20250812 19:58:32.252082 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:41739
--local_ip_for_outbound_sockets=127.16.93.194
--tserver_master_addrs=127.16.93.254:35295
--webserver_port=38701
--webserver_interface=127.16.93.194
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:32.257359 20148 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:32.258744 20148 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap complete.
I20250812 19:58:32.261055 20148 ts_tablet_manager.cc:1397] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.207s user 0.146s sys 0.056s
I20250812 19:58:32.285480 20148 raft_consensus.cc:357] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:32.287390 20148 raft_consensus.cc:738] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: FOLLOWER
I20250812 19:58:32.288537 20148 consensus_queue.cc:260] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:32.289289 20148 raft_consensus.cc:397] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:32.289686 20148 raft_consensus.cc:491] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:32.290141 20148 raft_consensus.cc:3058] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:58:32.300027 20148 raft_consensus.cc:513] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:32.300966 20148 leader_election.cc:304] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf17e5fbd8e148bfa3aa99f74803a8dc; no voters:
I20250812 19:58:32.303253 20148 leader_election.cc:290] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 19:58:32.303771 20261 raft_consensus.cc:2802] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Leader election won for term 2
I20250812 19:58:32.313454 20256 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:32.313936 20256 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:32.315402 20256 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:32.313076 20148 ts_tablet_manager.cc:1428] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.052s user 0.037s sys 0.012s
I20250812 19:58:32.317786 20261 raft_consensus.cc:695] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEADER]: Becoming Leader. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Running, Role: LEADER
I20250812 19:58:32.318619 20261 consensus_queue.cc:237] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:32.331035 20069 ts_manager.cc:194] Registered new tserver with Master: cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193:41115)
I20250812 19:58:32.335109 20069 catalog_manager.cc:5582] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } health_report { overall_health: HEALTHY } } }
I20250812 19:58:32.383006 20069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:60253
I20250812 19:58:32.387404 20256 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
W20250812 19:58:32.610149 20257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:32.610666 20257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:32.611181 20257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:32.642611 20257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:32.643476 20257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:58:32.678545 20257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:41739
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=38701
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:32.679888 20257 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:32.681443 20257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:32.694284 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:32.696295 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:33.999883 20277 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1305 milliseconds
W20250812 19:58:34.001942 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:34.004514 20257 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.310s user 0.495s sys 0.814s
W20250812 19:58:34.004909 20257 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.311s user 0.496s sys 0.814s
I20250812 19:58:34.005198 20257 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:34.006748 20257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:34.009658 20257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:34.011206 20257 hybrid_clock.cc:648] HybridClock initialized: now 1755028714011160 us; error 37 us; skew 500 ppm
I20250812 19:58:34.012439 20257 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:34.021549 20257 webserver.cc:480] Webserver started at http://127.16.93.194:38701/ using document root <none> and password file <none>
I20250812 19:58:34.023061 20257 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:34.023415 20257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:34.035697 20257 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20250812 19:58:34.042730 20286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:34.044189 20257 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250812 19:58:34.044636 20257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14f940533a744ceba9ebf397a1a45b8f"
format_stamp: "Formatted at 2025-08-12 19:58:19 on dist-test-slave-wz96"
I20250812 19:58:34.047537 20257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:34.138644 20257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:34.140151 20257 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:34.140565 20257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:34.143055 20257 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:34.148705 20293 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 19:58:34.156216 20257 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 19:58:34.156492 20257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250812 19:58:34.156813 20257 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 19:58:34.161511 20257 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 19:58:34.161697 20257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.002s
I20250812 19:58:34.162062 20293 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap starting.
I20250812 19:58:34.216338 20293 log.cc:826] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:34.306051 20293 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:34.307220 20293 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap complete.
I20250812 19:58:34.308754 20293 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Time spent bootstrapping tablet: real 0.147s user 0.098s sys 0.048s
I20250812 19:58:34.325875 20293 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:34.329355 20293 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Initialized, Role: FOLLOWER
I20250812 19:58:34.330188 20293 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:34.330683 20293 raft_consensus.cc:397] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:34.330947 20293 raft_consensus.cc:491] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:34.331244 20293 raft_consensus.cc:3058] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:58:34.337575 20293 raft_consensus.cc:513] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:34.338479 20293 leader_election.cc:304] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f940533a744ceba9ebf397a1a45b8f; no voters:
I20250812 19:58:34.338816 20257 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:41739
I20250812 19:58:34.338979 20400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:41739 every 8 connection(s)
I20250812 19:58:34.340445 20293 leader_election.cc:290] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 19:58:34.341156 20401 raft_consensus.cc:2802] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Leader election won for term 2
I20250812 19:58:34.342499 20257 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:58:34.344596 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20257
I20250812 19:58:34.344744 20401 raft_consensus.cc:695] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEADER]: Becoming Leader. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Running, Role: LEADER
I20250812 19:58:34.345018 20293 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Time spent starting tablet: real 0.036s user 0.035s sys 0.001s
I20250812 19:58:34.345580 20401 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } }
I20250812 19:58:34.347512 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:45977
--local_ip_for_outbound_sockets=127.16.93.195
--tserver_master_addrs=127.16.93.254:35295
--webserver_port=40469
--webserver_interface=127.16.93.195
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:34.377938 20404 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:34.378513 20404 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:34.379926 20404 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:34.384370 20069 ts_manager.cc:194] Registered new tserver with Master: 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
I20250812 19:58:34.385375 20069 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f reported cstate change: term changed from 0 to 2, leader changed from <none> to 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194), VOTER 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) added. New cstate: current_term: 2 leader_uuid: "14f940533a744ceba9ebf397a1a45b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } health_report { overall_health: HEALTHY } } }
I20250812 19:58:34.398669 20069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:53717
I20250812 19:58:34.405535 20404 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
I20250812 19:58:34.413890 20356 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:34.417896 20403 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } } }
I20250812 19:58:34.428265 20055 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21fd193e9c454edc98941ed66ec26d5a with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250812 19:58:34.430285 20290 consensus_peers.cc:489] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f -> Peer cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193:41115): Couldn't send request to peer cf17e5fbd8e148bfa3aa99f74803a8dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21fd193e9c454edc98941ed66ec26d5a. This is attempt 1: this message will repeat every 5th retry.
I20250812 19:58:34.431058 20068 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f reported cstate change: config changed from index -1 to 8, NON_VOTER cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) added. New cstate: current_term: 2 leader_uuid: "14f940533a744ceba9ebf397a1a45b8f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250812 19:58:34.438126 20068 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21fd193e9c454edc98941ed66ec26d5a with cas_config_opid_index 8: no extra replica candidate found for tablet 21fd193e9c454edc98941ed66ec26d5a (table TestTable [id=04bd440bf2214297934d556fc36b02b0]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250812 19:58:34.678174 20410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:34.678681 20410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:34.679183 20410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:34.710367 20410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:34.711220 20410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:58:34.746645 20410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:45977
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=40469
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:34.747959 20410 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:34.749483 20410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:34.761855 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:34.893471 20429 ts_tablet_manager.cc:927] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Initiating tablet copy from peer 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
I20250812 19:58:34.895869 20429 tablet_copy_client.cc:323] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Beginning tablet copy session from remote peer at address 127.16.93.194:41739
I20250812 19:58:34.909245 20376 tablet_copy_service.cc:140] P 14f940533a744ceba9ebf397a1a45b8f: Received BeginTabletCopySession request for tablet 21fd193e9c454edc98941ed66ec26d5a from peer cf17e5fbd8e148bfa3aa99f74803a8dc ({username='slave'} at 127.16.93.193:52919)
I20250812 19:58:34.909688 20376 tablet_copy_service.cc:161] P 14f940533a744ceba9ebf397a1a45b8f: Beginning new tablet copy session on tablet 21fd193e9c454edc98941ed66ec26d5a from peer cf17e5fbd8e148bfa3aa99f74803a8dc at {username='slave'} at 127.16.93.193:52919: session id = cf17e5fbd8e148bfa3aa99f74803a8dc-21fd193e9c454edc98941ed66ec26d5a
I20250812 19:58:34.914187 20376 tablet_copy_source_session.cc:215] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 19:58:34.917555 20429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21fd193e9c454edc98941ed66ec26d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:34.927892 20429 tablet_copy_client.cc:806] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Starting download of 0 data blocks...
I20250812 19:58:34.928412 20429 tablet_copy_client.cc:670] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Starting download of 1 WAL segments...
I20250812 19:58:34.932632 20429 tablet_copy_client.cc:538] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 19:58:34.938100 20429 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:35.008461 20429 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:35.009148 20429 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap complete.
I20250812 19:58:35.009670 20429 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.072s user 0.061s sys 0.010s
I20250812 19:58:35.011332 20429 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:35.011732 20429 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: LEARNER
I20250812 19:58:35.012187 20429 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:35.015115 20429 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250812 19:58:35.016542 20376 tablet_copy_service.cc:342] P 14f940533a744ceba9ebf397a1a45b8f: Request end of tablet copy session cf17e5fbd8e148bfa3aa99f74803a8dc-21fd193e9c454edc98941ed66ec26d5a received from {username='slave'} at 127.16.93.193:52919
I20250812 19:58:35.016978 20376 tablet_copy_service.cc:434] P 14f940533a744ceba9ebf397a1a45b8f: ending tablet copy session cf17e5fbd8e148bfa3aa99f74803a8dc-21fd193e9c454edc98941ed66ec26d5a on tablet 21fd193e9c454edc98941ed66ec26d5a with peer cf17e5fbd8e148bfa3aa99f74803a8dc
I20250812 19:58:35.292312 20211 raft_consensus.cc:1215] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250812 19:58:35.701231 20435 raft_consensus.cc:1062] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: attempting to promote NON_VOTER cf17e5fbd8e148bfa3aa99f74803a8dc to VOTER
I20250812 19:58:35.702646 20435 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:35.708248 20211 raft_consensus.cc:1273] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Refusing update from remote peer 14f940533a744ceba9ebf397a1a45b8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250812 19:58:35.709678 20434 consensus_queue.cc:1035] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250812 19:58:35.717451 20435 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:35.719740 20211 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:35.725997 20069 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f reported cstate change: config changed from index 8 to 9, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "14f940533a744ceba9ebf397a1a45b8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250812 19:58:34.769032 20426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:34.762394 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:35.949680 20425 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 19:58:35.949733 20410 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:35.954018 20410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:35.956717 20410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:35.958148 20410 hybrid_clock.cc:648] HybridClock initialized: now 1755028715958128 us; error 47 us; skew 500 ppm
I20250812 19:58:35.958937 20410 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:35.965543 20410 webserver.cc:480] Webserver started at http://127.16.93.195:40469/ using document root <none> and password file <none>
I20250812 19:58:35.966521 20410 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:35.966823 20410 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:35.974529 20410 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20250812 19:58:35.978967 20446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:35.980050 20410 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250812 19:58:35.980346 20410 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb685ac193094374b6df35b06da361e2"
format_stamp: "Formatted at 2025-08-12 19:58:21 on dist-test-slave-wz96"
I20250812 19:58:35.982183 20410 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:36.032332 20410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:36.033851 20410 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:36.034253 20410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:36.036834 20410 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:36.043119 20453 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 19:58:36.054136 20410 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 19:58:36.054383 20410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250812 19:58:36.054688 20410 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 19:58:36.059298 20410 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 19:58:36.059494 20410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250812 19:58:36.059975 20453 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap starting.
I20250812 19:58:36.119731 20453 log.cc:826] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:36.226466 20410 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:45977
I20250812 19:58:36.226550 20560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:45977 every 8 connection(s)
I20250812 19:58:36.229951 20410 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:58:36.236119 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20410
I20250812 19:58:36.258343 20453 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:36.259500 20453 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap complete.
I20250812 19:58:36.261138 20453 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Time spent bootstrapping tablet: real 0.202s user 0.146s sys 0.040s
I20250812 19:58:36.278278 20561 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:36.278803 20561 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:36.277324 20453 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:36.279691 20453 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb685ac193094374b6df35b06da361e2, State: Initialized, Role: FOLLOWER
I20250812 19:58:36.280057 20561 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:36.280431 20453 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:36.281132 20453 raft_consensus.cc:397] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:36.281383 20453 raft_consensus.cc:491] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:36.281689 20453 raft_consensus.cc:3058] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 1 FOLLOWER]: Advancing to term 2
I20250812 19:58:36.284288 20069 ts_manager.cc:194] Registered new tserver with Master: fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:36.287464 20069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:35313
I20250812 19:58:36.289134 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:36.289119 20453 raft_consensus.cc:513] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:36.289928 20453 leader_election.cc:304] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb685ac193094374b6df35b06da361e2; no voters:
I20250812 19:58:36.292281 20453 leader_election.cc:290] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 19:58:36.293218 20567 raft_consensus.cc:2802] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 19:58:36.293350 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:36.295586 20561 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
I20250812 19:58:36.295960 20453 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Time spent starting tablet: real 0.035s user 0.028s sys 0.005s
I20250812 19:58:36.297191 20567 raft_consensus.cc:695] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 LEADER]: Becoming Leader. State: Replica: fb685ac193094374b6df35b06da361e2, State: Running, Role: LEADER
W20250812 19:58:36.297115 16759 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250812 19:58:36.297978 20567 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } }
I20250812 19:58:36.308629 20069 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 reported cstate change: term changed from 0 to 2, leader changed from <none> to fb685ac193094374b6df35b06da361e2 (127.16.93.195), VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) added. New cstate: current_term: 2 leader_uuid: "fb685ac193094374b6df35b06da361e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: HEALTHY } } }
I20250812 19:58:36.331274 20516 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } }
I20250812 19:58:36.334952 20569 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } }
I20250812 19:58:36.356460 20056 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c547a53cb7c647ad8ed0762e9470b5ee with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250812 19:58:36.357791 20356 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: NON_VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: true } }
W20250812 19:58:36.359393 20449 consensus_peers.cc:489] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 -> Peer 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739): Couldn't send request to peer 14f940533a744ceba9ebf397a1a45b8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c547a53cb7c647ad8ed0762e9470b5ee. This is attempt 1: this message will repeat every 5th retry.
I20250812 19:58:36.360426 20069 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 reported cstate change: config changed from index -1 to 8, NON_VOTER 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) added. New cstate: current_term: 2 leader_uuid: "fb685ac193094374b6df35b06da361e2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250812 19:58:36.366204 20211 raft_consensus.cc:1273] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Refusing update from remote peer 14f940533a744ceba9ebf397a1a45b8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250812 19:58:36.367555 20434 consensus_queue.cc:1035] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250812 19:58:36.371930 20516 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
W20250812 19:58:36.374389 20290 consensus_peers.cc:489] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f -> Peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Couldn't send request to peer fb685ac193094374b6df35b06da361e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21fd193e9c454edc98941ed66ec26d5a. This is attempt 1: this message will repeat every 5th retry.
I20250812 19:58:36.375083 20579 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: NON_VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: true } } }
I20250812 19:58:36.376541 20211 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: NON_VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: true } } }
I20250812 19:58:36.376649 20569 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } } }
I20250812 19:58:36.386312 20055 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21fd193e9c454edc98941ed66ec26d5a with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250812 19:58:36.386586 20056 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c547a53cb7c647ad8ed0762e9470b5ee with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250812 19:58:36.387809 20449 consensus_peers.cc:489] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 -> Peer 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739): Couldn't send request to peer 14f940533a744ceba9ebf397a1a45b8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c547a53cb7c647ad8ed0762e9470b5ee. This is attempt 1: this message will repeat every 5th retry.
I20250812 19:58:36.389695 20069 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 reported cstate change: config changed from index 8 to 9, NON_VOTER cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) added. New cstate: current_term: 2 leader_uuid: "fb685ac193094374b6df35b06da361e2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250812 19:58:36.390067 20068 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f reported cstate change: config changed from index 9 to 10, NON_VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) added. New cstate: current_term: 2 leader_uuid: "14f940533a744ceba9ebf397a1a45b8f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: NON_VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250812 19:58:36.398046 20450 consensus_peers.cc:489] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 -> Peer cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193:41115): Couldn't send request to peer cf17e5fbd8e148bfa3aa99f74803a8dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c547a53cb7c647ad8ed0762e9470b5ee. This is attempt 1: this message will repeat every 5th retry.
I20250812 19:58:36.735811 20054 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21fd193e9c454edc98941ed66ec26d5a with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250812 19:58:36.809582 20582 ts_tablet_manager.cc:927] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Initiating tablet copy from peer 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
I20250812 19:58:36.811444 20582 tablet_copy_client.cc:323] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.93.194:41739
I20250812 19:58:36.812883 20376 tablet_copy_service.cc:140] P 14f940533a744ceba9ebf397a1a45b8f: Received BeginTabletCopySession request for tablet 21fd193e9c454edc98941ed66ec26d5a from peer fb685ac193094374b6df35b06da361e2 ({username='slave'} at 127.16.93.195:57227)
I20250812 19:58:36.813244 20376 tablet_copy_service.cc:161] P 14f940533a744ceba9ebf397a1a45b8f: Beginning new tablet copy session on tablet 21fd193e9c454edc98941ed66ec26d5a from peer fb685ac193094374b6df35b06da361e2 at {username='slave'} at 127.16.93.195:57227: session id = fb685ac193094374b6df35b06da361e2-21fd193e9c454edc98941ed66ec26d5a
I20250812 19:58:36.817458 20376 tablet_copy_source_session.cc:215] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 19:58:36.820194 20582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21fd193e9c454edc98941ed66ec26d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:36.832197 20582 tablet_copy_client.cc:806] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: tablet copy: Starting download of 0 data blocks...
I20250812 19:58:36.832718 20582 tablet_copy_client.cc:670] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: tablet copy: Starting download of 1 WAL segments...
I20250812 19:58:36.836089 20582 tablet_copy_client.cc:538] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 19:58:36.841217 20582 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Bootstrap starting.
I20250812 19:58:36.909834 20586 ts_tablet_manager.cc:927] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Initiating tablet copy from peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:36.911501 20586 tablet_copy_client.cc:323] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.93.195:45977
I20250812 19:58:36.912992 20536 tablet_copy_service.cc:140] P fb685ac193094374b6df35b06da361e2: Received BeginTabletCopySession request for tablet c547a53cb7c647ad8ed0762e9470b5ee from peer 14f940533a744ceba9ebf397a1a45b8f ({username='slave'} at 127.16.93.194:51239)
I20250812 19:58:36.913321 20582 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:36.913558 20536 tablet_copy_service.cc:161] P fb685ac193094374b6df35b06da361e2: Beginning new tablet copy session on tablet c547a53cb7c647ad8ed0762e9470b5ee from peer 14f940533a744ceba9ebf397a1a45b8f at {username='slave'} at 127.16.93.194:51239: session id = 14f940533a744ceba9ebf397a1a45b8f-c547a53cb7c647ad8ed0762e9470b5ee
I20250812 19:58:36.914040 20582 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Bootstrap complete.
I20250812 19:58:36.914597 20582 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Time spent bootstrapping tablet: real 0.074s user 0.067s sys 0.005s
I20250812 19:58:36.916793 20582 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: NON_VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: true } }
I20250812 19:58:36.917447 20582 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fb685ac193094374b6df35b06da361e2, State: Initialized, Role: LEARNER
I20250812 19:58:36.917948 20582 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: NON_VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: true } }
I20250812 19:58:36.919817 20582 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250812 19:58:36.920423 20536 tablet_copy_source_session.cc:215] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 19:58:36.921253 20376 tablet_copy_service.cc:342] P 14f940533a744ceba9ebf397a1a45b8f: Request end of tablet copy session fb685ac193094374b6df35b06da361e2-21fd193e9c454edc98941ed66ec26d5a received from {username='slave'} at 127.16.93.195:57227
I20250812 19:58:36.921623 20376 tablet_copy_service.cc:434] P 14f940533a744ceba9ebf397a1a45b8f: ending tablet copy session fb685ac193094374b6df35b06da361e2-21fd193e9c454edc98941ed66ec26d5a on tablet 21fd193e9c454edc98941ed66ec26d5a with peer fb685ac193094374b6df35b06da361e2
I20250812 19:58:36.924373 20586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c547a53cb7c647ad8ed0762e9470b5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:36.934265 20586 tablet_copy_client.cc:806] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: tablet copy: Starting download of 0 data blocks...
I20250812 19:58:36.934785 20586 tablet_copy_client.cc:670] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: tablet copy: Starting download of 1 WAL segments...
I20250812 19:58:36.938388 20586 tablet_copy_client.cc:538] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 19:58:36.943614 20586 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap starting.
I20250812 19:58:36.959986 20589 ts_tablet_manager.cc:927] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Initiating tablet copy from peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:36.961429 20589 tablet_copy_client.cc:323] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Beginning tablet copy session from remote peer at address 127.16.93.195:45977
I20250812 19:58:36.970619 20536 tablet_copy_service.cc:140] P fb685ac193094374b6df35b06da361e2: Received BeginTabletCopySession request for tablet c547a53cb7c647ad8ed0762e9470b5ee from peer cf17e5fbd8e148bfa3aa99f74803a8dc ({username='slave'} at 127.16.93.193:47165)
I20250812 19:58:36.971056 20536 tablet_copy_service.cc:161] P fb685ac193094374b6df35b06da361e2: Beginning new tablet copy session on tablet c547a53cb7c647ad8ed0762e9470b5ee from peer cf17e5fbd8e148bfa3aa99f74803a8dc at {username='slave'} at 127.16.93.193:47165: session id = cf17e5fbd8e148bfa3aa99f74803a8dc-c547a53cb7c647ad8ed0762e9470b5ee
I20250812 19:58:36.975247 20536 tablet_copy_source_session.cc:215] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 19:58:36.977754 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c547a53cb7c647ad8ed0762e9470b5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:58:36.993018 20589 tablet_copy_client.cc:806] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Starting download of 0 data blocks...
I20250812 19:58:36.993534 20589 tablet_copy_client.cc:670] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Starting download of 1 WAL segments...
I20250812 19:58:36.997502 20589 tablet_copy_client.cc:538] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 19:58:37.005894 20589 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:37.028244 20586 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:37.028875 20586 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap complete.
I20250812 19:58:37.029304 20586 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Time spent bootstrapping tablet: real 0.086s user 0.071s sys 0.016s
I20250812 19:58:37.030956 20586 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:37.031409 20586 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Initialized, Role: LEARNER
I20250812 19:58:37.031823 20586 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:37.033421 20586 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250812 19:58:37.035079 20536 tablet_copy_service.cc:342] P fb685ac193094374b6df35b06da361e2: Request end of tablet copy session 14f940533a744ceba9ebf397a1a45b8f-c547a53cb7c647ad8ed0762e9470b5ee received from {username='slave'} at 127.16.93.194:51239
I20250812 19:58:37.035522 20536 tablet_copy_service.cc:434] P fb685ac193094374b6df35b06da361e2: ending tablet copy session 14f940533a744ceba9ebf397a1a45b8f-c547a53cb7c647ad8ed0762e9470b5ee on tablet c547a53cb7c647ad8ed0762e9470b5ee with peer 14f940533a744ceba9ebf397a1a45b8f
I20250812 19:58:37.079613 20589 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:37.080227 20589 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap complete.
I20250812 19:58:37.080708 20589 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.075s user 0.075s sys 0.001s
I20250812 19:58:37.082257 20589 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:37.082729 20589 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: LEARNER
I20250812 19:58:37.083124 20589 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: NON_VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:37.085721 20589 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20250812 19:58:37.087312 20536 tablet_copy_service.cc:342] P fb685ac193094374b6df35b06da361e2: Request end of tablet copy session cf17e5fbd8e148bfa3aa99f74803a8dc-c547a53cb7c647ad8ed0762e9470b5ee received from {username='slave'} at 127.16.93.193:47165
I20250812 19:58:37.087627 20536 tablet_copy_service.cc:434] P fb685ac193094374b6df35b06da361e2: ending tablet copy session cf17e5fbd8e148bfa3aa99f74803a8dc-c547a53cb7c647ad8ed0762e9470b5ee on tablet c547a53cb7c647ad8ed0762e9470b5ee with peer cf17e5fbd8e148bfa3aa99f74803a8dc
I20250812 19:58:37.223107 20516 raft_consensus.cc:1215] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250812 19:58:37.301712 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cf17e5fbd8e148bfa3aa99f74803a8dc to finish bootstrapping
I20250812 19:58:37.317420 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 14f940533a744ceba9ebf397a1a45b8f to finish bootstrapping
I20250812 19:58:37.329365 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fb685ac193094374b6df35b06da361e2 to finish bootstrapping
I20250812 19:58:37.505092 20211 raft_consensus.cc:1215] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250812 19:58:37.514660 20356 raft_consensus.cc:1215] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250812 19:58:37.600265 20191 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 19:58:37.600261 20336 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 19:58:37.602751 20496 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 19:58:37.678180 20595 raft_consensus.cc:1062] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: attempting to promote NON_VOTER fb685ac193094374b6df35b06da361e2 to VOTER
I20250812 19:58:37.685294 20595 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:37.700131 20516 raft_consensus.cc:1273] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 LEARNER]: Refusing update from remote peer 14f940533a744ceba9ebf397a1a45b8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250812 19:58:37.701810 20596 consensus_queue.cc:1035] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250812 19:58:37.711064 20211 raft_consensus.cc:1273] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Refusing update from remote peer 14f940533a744ceba9ebf397a1a45b8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250812 19:58:37.714018 20596 consensus_queue.cc:1035] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250812 19:58:37.732975 20211 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, fb685ac193094374b6df35b06da361e2 (127.16.93.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } } }
I20250812 19:58:37.742844 20595 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, fb685ac193094374b6df35b06da361e2 (127.16.93.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } } }
I20250812 19:58:37.753944 20068 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: config changed from index 10 to 11, fb685ac193094374b6df35b06da361e2 (127.16.93.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "14f940533a744ceba9ebf397a1a45b8f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } } }
I20250812 19:58:37.770301 20516 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, fb685ac193094374b6df35b06da361e2 (127.16.93.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
fd532d514cb443c18faf85b14f1a5eb7 | 127.16.93.254:35295 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.93.212:34673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
14f940533a744ceba9ebf397a1a45b8f | 127.16.93.194:41739 | HEALTHY | <none> | 1 | 0
cf17e5fbd8e148bfa3aa99f74803a8dc | 127.16.93.193:41115 | HEALTHY | <none> | 1 | 0
fb685ac193094374b6df35b06da361e2 | 127.16.93.195:45977 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.16.93.193 | experimental | 127.16.93.193:41115
local_ip_for_outbound_sockets | 127.16.93.194 | experimental | 127.16.93.194:41739
local_ip_for_outbound_sockets | 127.16.93.195 | experimental | 127.16.93.195:45977
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.93.193:41115
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.93.194:41739
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.93.195:45977
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.93.212:34673 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250812 19:58:37.849568 16759 log_verifier.cc:126] Checking tablet 0b297f6787f94b618af8d9c680ffd066
I20250812 19:58:37.879812 16759 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0b297f6787f94b618af8d9c680ffd066
I20250812 19:58:37.880053 16759 log_verifier.cc:126] Checking tablet 21fd193e9c454edc98941ed66ec26d5a
I20250812 19:58:37.897363 20608 raft_consensus.cc:1062] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: attempting to promote NON_VOTER 14f940533a744ceba9ebf397a1a45b8f to VOTER
I20250812 19:58:37.898819 20608 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } }
I20250812 19:58:37.904858 20211 raft_consensus.cc:1273] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Refusing update from remote peer fb685ac193094374b6df35b06da361e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250812 19:58:37.904942 20356 raft_consensus.cc:1273] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 LEARNER]: Refusing update from remote peer fb685ac193094374b6df35b06da361e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250812 19:58:37.906189 20608 consensus_queue.cc:1035] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250812 19:58:37.907001 20610 consensus_queue.cc:1035] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250812 19:58:37.915112 20610 raft_consensus.cc:1025] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 LEADER]: attempt to promote peer cf17e5fbd8e148bfa3aa99f74803a8dc: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250812 19:58:37.916982 20608 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } } }
I20250812 19:58:37.918519 20356 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } } }
I20250812 19:58:37.929877 20069 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 reported cstate change: config changed from index 9 to 10, 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fb685ac193094374b6df35b06da361e2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250812 19:58:37.930956 20211 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: NON_VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: true } } }
I20250812 19:58:37.941823 20585 raft_consensus.cc:1062] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: attempting to promote NON_VOTER cf17e5fbd8e148bfa3aa99f74803a8dc to VOTER
I20250812 19:58:37.943786 20585 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:37.949527 20211 raft_consensus.cc:1273] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 LEARNER]: Refusing update from remote peer fb685ac193094374b6df35b06da361e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250812 19:58:37.949672 20356 raft_consensus.cc:1273] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Refusing update from remote peer fb685ac193094374b6df35b06da361e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250812 19:58:37.950769 20610 consensus_queue.cc:1035] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250812 19:58:37.951467 20608 consensus_queue.cc:1035] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250812 19:58:37.957347 20585 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:37.959264 20211 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:37.959209 20356 raft_consensus.cc:2953] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:37.970578 20067 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 reported cstate change: config changed from index 10 to 11, cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fb685ac193094374b6df35b06da361e2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250812 19:58:37.987866 16759 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 21fd193e9c454edc98941ed66ec26d5a
I20250812 19:58:37.988088 16759 log_verifier.cc:126] Checking tablet c547a53cb7c647ad8ed0762e9470b5ee
I20250812 19:58:38.070351 16759 log_verifier.cc:177] Verified matching terms for 11 ops in tablet c547a53cb7c647ad8ed0762e9470b5ee
I20250812 19:58:38.070791 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20036
I20250812 19:58:38.100178 16759 minidump.cc:252] Setting minidump size limit to 20M
I20250812 19:58:38.101501 16759 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:38.102828 16759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:38.115398 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:38.115674 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:38.116999 20641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:38.117061 16759 server_base.cc:1047] running on GCE node
I20250812 19:58:38.214766 16759 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250812 19:58:38.214980 16759 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250812 19:58:38.215113 16759 hybrid_clock.cc:648] HybridClock initialized: now 1755028718215101 us; error 0 us; skew 500 ppm
I20250812 19:58:38.215704 16759 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:38.218739 16759 webserver.cc:480] Webserver started at http://0.0.0.0:37587/ using document root <none> and password file <none>
I20250812 19:58:38.219508 16759 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:38.219677 16759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:38.225229 16759 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250812 19:58:38.229688 20646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:38.230597 16759 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 19:58:38.230891 16759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fd532d514cb443c18faf85b14f1a5eb7"
format_stamp: "Formatted at 2025-08-12 19:58:15 on dist-test-slave-wz96"
I20250812 19:58:38.232517 16759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:38.247727 16759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:38.249490 16759 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:38.250020 16759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:38.262243 16759 sys_catalog.cc:263] Verifying existing consensus state
W20250812 19:58:38.267673 16759 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.16.93.254:35295 exists but no master address supplied!
I20250812 19:58:38.269997 16759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap starting.
I20250812 19:58:38.318347 16759 log.cc:826] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:38.392639 16759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:38.393424 16759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap complete.
I20250812 19:58:38.406174 16759 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:38.406754 16759 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Initialized, Role: FOLLOWER
I20250812 19:58:38.407470 16759 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:38.407922 16759 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:38.408177 16759 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:38.408521 16759 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 3 FOLLOWER]: Advancing to term 4
I20250812 19:58:38.413825 16759 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:38.414481 16759 leader_election.cc:304] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd532d514cb443c18faf85b14f1a5eb7; no voters:
I20250812 19:58:38.415544 16759 leader_election.cc:290] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250812 19:58:38.415791 20653 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 4 FOLLOWER]: Leader election won for term 4
I20250812 19:58:38.417781 20653 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 4 LEADER]: Becoming Leader. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Running, Role: LEADER
I20250812 19:58:38.418567 20653 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:38.425412 20654 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:38.425865 20655 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd532d514cb443c18faf85b14f1a5eb7. Latest consensus state: current_term: 4 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:38.426049 20654 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:38.426313 20655 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:38.452766 16759 tablet_replica.cc:331] stopping tablet replica
I20250812 19:58:38.453300 16759 raft_consensus.cc:2241] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 4 LEADER]: Raft consensus shutting down.
I20250812 19:58:38.453723 16759 raft_consensus.cc:2270] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250812 19:58:38.455893 16759 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250812 19:58:38.456358 16759 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250812 19:58:38.486107 16759 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250812 19:58:38.989717 20404 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:35295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:35295: connect: Connection refused (error 111)
W20250812 19:58:39.003041 20256 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:35295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:35295: connect: Connection refused (error 111)
W20250812 19:58:39.008679 20561 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:35295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:35295: connect: Connection refused (error 111)
I20250812 19:58:43.789430 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20106
I20250812 19:58:43.817126 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20257
I20250812 19:58:43.843715 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20410
I20250812 19:58:43.877987 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35295
--webserver_interface=127.16.93.254
--webserver_port=45775
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:35295 with env {}
W20250812 19:58:44.190752 20733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:44.191356 20733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:44.191821 20733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:44.222822 20733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:58:44.223173 20733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:44.223441 20733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:58:44.223685 20733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:58:44.258173 20733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:35295
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=45775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:44.259493 20733 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:44.261114 20733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:44.270977 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:44.272747 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:45.465924 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:45.468919 20741 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1191 milliseconds
I20250812 19:58:45.469043 20733 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:45.470321 20733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:45.473573 20733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:45.475029 20733 hybrid_clock.cc:648] HybridClock initialized: now 1755028725474986 us; error 64 us; skew 500 ppm
I20250812 19:58:45.475868 20733 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:45.482342 20733 webserver.cc:480] Webserver started at http://127.16.93.254:45775/ using document root <none> and password file <none>
I20250812 19:58:45.483328 20733 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:45.483557 20733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:45.491768 20733 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250812 19:58:45.496480 20749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:45.497571 20733 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250812 19:58:45.497905 20733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fd532d514cb443c18faf85b14f1a5eb7"
format_stamp: "Formatted at 2025-08-12 19:58:15 on dist-test-slave-wz96"
I20250812 19:58:45.499781 20733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:45.557811 20733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:45.559196 20733 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:45.559609 20733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:45.628801 20733 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:35295
I20250812 19:58:45.628887 20800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:35295 every 8 connection(s)
I20250812 19:58:45.631634 20733 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:58:45.636037 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20733
I20250812 19:58:45.637527 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:41115
--local_ip_for_outbound_sockets=127.16.93.193
--tserver_master_addrs=127.16.93.254:35295
--webserver_port=41145
--webserver_interface=127.16.93.193
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:45.642374 20801 sys_catalog.cc:263] Verifying existing consensus state
I20250812 19:58:45.647040 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap starting.
I20250812 19:58:45.656824 20801 log.cc:826] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:45.732556 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:45.733374 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Bootstrap complete.
I20250812 19:58:45.751323 20801 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:45.753379 20801 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Initialized, Role: FOLLOWER
I20250812 19:58:45.754190 20801 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:45.754665 20801 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:45.754914 20801 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:45.755210 20801 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 5 FOLLOWER]: Advancing to term 6
I20250812 19:58:45.760237 20801 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:45.760893 20801 leader_election.cc:304] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd532d514cb443c18faf85b14f1a5eb7; no voters:
I20250812 19:58:45.762998 20801 leader_election.cc:290] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250812 19:58:45.763397 20805 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 6 FOLLOWER]: Leader election won for term 6
I20250812 19:58:45.766637 20805 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [term 6 LEADER]: Becoming Leader. State: Replica: fd532d514cb443c18faf85b14f1a5eb7, State: Running, Role: LEADER
I20250812 19:58:45.767520 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } }
I20250812 19:58:45.767913 20801 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:58:45.777886 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:45.778620 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:45.781528 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd532d514cb443c18faf85b14f1a5eb7. Latest consensus state: current_term: 6 leader_uuid: "fd532d514cb443c18faf85b14f1a5eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd532d514cb443c18faf85b14f1a5eb7" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35295 } } }
I20250812 19:58:45.782255 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7 [sys.catalog]: This master's current role is: LEADER
I20250812 19:58:45.789572 20812 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:58:45.806241 20812 catalog_manager.cc:671] Loaded metadata for table TestTable [id=10cfd055e63e4937b0d9230df9d58691]
I20250812 19:58:45.808619 20812 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=4958118b9eea4eb8a7ad1badb5e9dd61]
I20250812 19:58:45.810595 20812 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=f312994f95cf43768326d77ca3a13033]
I20250812 19:58:45.822374 20812 tablet_loader.cc:96] loaded metadata for tablet 0b297f6787f94b618af8d9c680ffd066 (table TestTable2 [id=4958118b9eea4eb8a7ad1badb5e9dd61])
I20250812 19:58:45.824088 20812 tablet_loader.cc:96] loaded metadata for tablet 21fd193e9c454edc98941ed66ec26d5a (table TestTable [id=10cfd055e63e4937b0d9230df9d58691])
I20250812 19:58:45.825963 20812 tablet_loader.cc:96] loaded metadata for tablet c547a53cb7c647ad8ed0762e9470b5ee (table TestTable1 [id=f312994f95cf43768326d77ca3a13033])
I20250812 19:58:45.827782 20812 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:58:45.834874 20812 catalog_manager.cc:1261] Loaded cluster ID: bb833e26fffb4fe9bbcca16f5584a760
I20250812 19:58:45.835247 20812 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:58:45.843313 20822 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250812 19:58:45.845067 20812 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:58:45.848460 20812 catalog_manager.cc:5966] T 00000000000000000000000000000000 P fd532d514cb443c18faf85b14f1a5eb7: Loaded TSK: 0
I20250812 19:58:45.849550 20812 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 19:58:45.983776 20803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:45.984288 20803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:45.984880 20803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:46.017210 20803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:46.018070 20803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:58:46.054373 20803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:41115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=41145
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:46.055742 20803 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:46.057286 20803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:46.070235 20828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:46.074621 20831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:46.075910 20803 server_base.cc:1047] running on GCE node
W20250812 19:58:46.075395 20829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:47.245321 20803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:47.248070 20803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:47.249524 20803 hybrid_clock.cc:648] HybridClock initialized: now 1755028727249462 us; error 78 us; skew 500 ppm
I20250812 19:58:47.250344 20803 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:47.257426 20803 webserver.cc:480] Webserver started at http://127.16.93.193:41145/ using document root <none> and password file <none>
I20250812 19:58:47.258343 20803 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:47.258559 20803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:47.266605 20803 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20250812 19:58:47.271669 20838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:47.272962 20803 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250812 19:58:47.273289 20803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc"
format_stamp: "Formatted at 2025-08-12 19:58:17 on dist-test-slave-wz96"
I20250812 19:58:47.275233 20803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:47.333752 20803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:47.335254 20803 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:47.335695 20803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:47.338778 20803 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:47.345178 20845 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250812 19:58:47.360818 20803 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250812 19:58:47.361050 20803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.003s
I20250812 19:58:47.361352 20803 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250812 19:58:47.366570 20845 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:47.376631 20803 ts_tablet_manager.cc:610] Registered 3 tablets
I20250812 19:58:47.376897 20803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.016s user 0.011s sys 0.005s
I20250812 19:58:47.440953 20845 log.cc:826] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:47.570015 20845 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:47.571225 20845 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap complete.
I20250812 19:58:47.572952 20845 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.207s user 0.167s sys 0.035s
I20250812 19:58:47.575718 20803 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:41115
I20250812 19:58:47.575923 20952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:41115 every 8 connection(s)
I20250812 19:58:47.579242 20803 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:58:47.587002 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20803
I20250812 19:58:47.590586 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:41739
--local_ip_for_outbound_sockets=127.16.93.194
--tserver_master_addrs=127.16.93.254:35295
--webserver_port=38701
--webserver_interface=127.16.93.194
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:47.593755 20845 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:47.603082 20845 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: FOLLOWER
I20250812 19:58:47.604204 20845 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:47.614329 20845 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.041s user 0.028s sys 0.004s
I20250812 19:58:47.615307 20845 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:47.633934 20953 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:47.634631 20953 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:47.636322 20953 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:47.642974 20766 ts_manager.cc:194] Registered new tserver with Master: cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193:41115)
I20250812 19:58:47.648253 20766 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) added, VOTER cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193) added, VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } } }
I20250812 19:58:47.735939 20766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:43685
I20250812 19:58:47.740571 20953 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
I20250812 19:58:47.759954 20845 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:47.760921 20845 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap complete.
I20250812 19:58:47.762055 20845 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.147s user 0.115s sys 0.028s
I20250812 19:58:47.763589 20845 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:47.764052 20845 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: FOLLOWER
I20250812 19:58:47.764559 20845 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:47.766289 20845 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250812 19:58:47.766907 20845 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap starting.
I20250812 19:58:47.847565 20845 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:47.848229 20845 tablet_bootstrap.cc:492] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Bootstrap complete.
I20250812 19:58:47.849445 20845 ts_tablet_manager.cc:1397] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent bootstrapping tablet: real 0.083s user 0.062s sys 0.020s
I20250812 19:58:47.850909 20845 raft_consensus.cc:357] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:47.851263 20845 raft_consensus.cc:738] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Initialized, Role: FOLLOWER
I20250812 19:58:47.851716 20845 consensus_queue.cc:260] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:47.852094 20845 raft_consensus.cc:397] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:58:47.852334 20845 raft_consensus.cc:491] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:58:47.852644 20845 raft_consensus.cc:3058] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:58:47.857777 20845 raft_consensus.cc:513] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:47.858635 20845 leader_election.cc:304] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf17e5fbd8e148bfa3aa99f74803a8dc; no voters:
I20250812 19:58:47.859076 20845 leader_election.cc:290] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: Requested vote from peers
I20250812 19:58:47.859338 20958 raft_consensus.cc:2802] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 FOLLOWER]: Leader election won for term 3
I20250812 19:58:47.862524 20845 ts_tablet_manager.cc:1428] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20250812 19:58:47.862970 20958 raft_consensus.cc:695] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 LEADER]: Becoming Leader. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Running, Role: LEADER
I20250812 19:58:47.863797 20958 consensus_queue.cc:237] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } }
I20250812 19:58:47.876058 20766 catalog_manager.cc:5582] T 0b297f6787f94b618af8d9c680ffd066 P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } health_report { overall_health: HEALTHY } } }
W20250812 19:58:47.960189 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:47.960695 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:47.961153 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:47.992317 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:47.993127 20957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:58:48.028564 20957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:41739
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=38701
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:48.029815 20957 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:48.031333 20957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:48.042569 20974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:49.109483 20981 raft_consensus.cc:491] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:58:49.110287 20981 raft_consensus.cc:513] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:49.124979 20981 leader_election.cc:290] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977), 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
W20250812 19:58:49.125706 20842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
I20250812 19:58:49.118356 20980 raft_consensus.cc:491] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:58:49.130375 20980 raft_consensus.cc:513] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
W20250812 19:58:49.132426 20842 leader_election.cc:336] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
W20250812 19:58:49.140049 20842 leader_election.cc:336] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
I20250812 19:58:49.141266 20980 leader_election.cc:290] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739), fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
W20250812 19:58:49.146955 20841 leader_election.cc:336] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739): Network error: Client connection negotiation failed: client connection to 127.16.93.194:41739: connect: Connection refused (error 111)
I20250812 19:58:49.147629 20841 leader_election.cc:304] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf17e5fbd8e148bfa3aa99f74803a8dc; no voters: 14f940533a744ceba9ebf397a1a45b8f, fb685ac193094374b6df35b06da361e2
I20250812 19:58:49.149636 20980 raft_consensus.cc:2747] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250812 19:58:49.153019 20841 leader_election.cc:336] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739): Network error: Client connection negotiation failed: client connection to 127.16.93.194:41739: connect: Connection refused (error 111)
I20250812 19:58:49.153661 20841 leader_election.cc:304] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf17e5fbd8e148bfa3aa99f74803a8dc; no voters: 14f940533a744ceba9ebf397a1a45b8f, fb685ac193094374b6df35b06da361e2
I20250812 19:58:49.154616 20980 raft_consensus.cc:2747] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250812 19:58:49.448680 20973 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 20957
W20250812 19:58:49.784505 20957 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.742s user 0.678s sys 1.057s
W20250812 19:58:49.784863 20957 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.742s user 0.679s sys 1.057s
W20250812 19:58:48.043406 20975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:49.786657 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:49.789599 20976 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1743 milliseconds
I20250812 19:58:49.789659 20957 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:49.790782 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:49.792758 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:49.794100 20957 hybrid_clock.cc:648] HybridClock initialized: now 1755028729794060 us; error 46 us; skew 500 ppm
I20250812 19:58:49.794849 20957 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:49.800555 20957 webserver.cc:480] Webserver started at http://127.16.93.194:38701/ using document root <none> and password file <none>
I20250812 19:58:49.801419 20957 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:49.801626 20957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:49.809062 20957 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250812 19:58:49.813433 20989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:49.814406 20957 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 19:58:49.814711 20957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14f940533a744ceba9ebf397a1a45b8f"
format_stamp: "Formatted at 2025-08-12 19:58:19 on dist-test-slave-wz96"
I20250812 19:58:49.816707 20957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:49.870127 20957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:49.871585 20957 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:49.872031 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:49.874467 20957 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:49.879729 20996 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250812 19:58:49.891005 20957 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250812 19:58:49.891253 20957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250812 19:58:49.891520 20957 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250812 19:58:49.896559 20996 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap starting.
I20250812 19:58:49.900092 20957 ts_tablet_manager.cc:610] Registered 2 tablets
I20250812 19:58:49.900305 20957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.003s sys 0.003s
I20250812 19:58:49.949259 20996 log.cc:826] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:50.061507 20957 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:41739
I20250812 19:58:50.061654 21103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:41739 every 8 connection(s)
I20250812 19:58:50.065016 20957 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:58:50.075544 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 20957
I20250812 19:58:50.077248 20996 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:50.077579 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:45977
--local_ip_for_outbound_sockets=127.16.93.195
--tserver_master_addrs=127.16.93.254:35295
--webserver_port=40469
--webserver_interface=127.16.93.195
--builtin_ntp_servers=127.16.93.212:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 19:58:50.078079 20996 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap complete.
I20250812 19:58:50.079352 20996 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Time spent bootstrapping tablet: real 0.183s user 0.147s sys 0.032s
I20250812 19:58:50.097893 21104 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:50.098323 21104 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:50.095957 20996 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:50.099313 21104 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:50.099550 20996 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Initialized, Role: FOLLOWER
I20250812 19:58:50.100469 20996 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:50.105499 20996 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Time spent starting tablet: real 0.026s user 0.027s sys 0.000s
I20250812 19:58:50.106398 20996 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap starting.
I20250812 19:58:50.109030 20766 ts_manager.cc:194] Registered new tserver with Master: 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
I20250812 19:58:50.112509 20766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:47281
I20250812 19:58:50.115567 21104 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
I20250812 19:58:50.199069 20996 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:50.199764 20996 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Bootstrap complete.
I20250812 19:58:50.200924 20996 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Time spent bootstrapping tablet: real 0.095s user 0.074s sys 0.020s
I20250812 19:58:50.202433 20996 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:50.202859 20996 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f940533a744ceba9ebf397a1a45b8f, State: Initialized, Role: FOLLOWER
I20250812 19:58:50.203356 20996 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:50.204959 20996 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250812 19:58:50.398980 21108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:58:50.399488 21108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:58:50.399979 21108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:58:50.430902 21108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:58:50.431720 21108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:58:50.466938 21108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:45977
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=40469
--tserver_master_addrs=127.16.93.254:35295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:58:50.468309 21108 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:58:50.469949 21108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:58:50.481993 21116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:58:50.751950 21123 raft_consensus.cc:491] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:58:50.753954 21123 raft_consensus.cc:513] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:50.753336 21122 raft_consensus.cc:491] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:58:50.764451 21122 raft_consensus.cc:513] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:50.769973 21123 leader_election.cc:290] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739), fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:50.781558 21122 leader_election.cc:290] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977), 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
W20250812 19:58:50.798777 20842 leader_election.cc:336] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
W20250812 19:58:50.803122 20842 leader_election.cc:336] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
I20250812 19:58:50.805935 21059 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" candidate_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "14f940533a744ceba9ebf397a1a45b8f" is_pre_election: true
I20250812 19:58:50.807070 21059 raft_consensus.cc:2466] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf17e5fbd8e148bfa3aa99f74803a8dc in term 2.
I20250812 19:58:50.809166 21058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c547a53cb7c647ad8ed0762e9470b5ee" candidate_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "14f940533a744ceba9ebf397a1a45b8f" is_pre_election: true
I20250812 19:58:50.809830 21058 raft_consensus.cc:2466] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf17e5fbd8e148bfa3aa99f74803a8dc in term 2.
I20250812 19:58:50.811620 20841 leader_election.cc:304] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f940533a744ceba9ebf397a1a45b8f, cf17e5fbd8e148bfa3aa99f74803a8dc; no voters: fb685ac193094374b6df35b06da361e2
I20250812 19:58:50.813097 20841 leader_election.cc:304] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f940533a744ceba9ebf397a1a45b8f, cf17e5fbd8e148bfa3aa99f74803a8dc; no voters: fb685ac193094374b6df35b06da361e2
I20250812 19:58:50.814046 21123 raft_consensus.cc:2802] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250812 19:58:50.814332 21122 raft_consensus.cc:2802] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250812 19:58:50.814541 21123 raft_consensus.cc:491] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:58:50.814805 21122 raft_consensus.cc:491] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:58:50.815023 21123 raft_consensus.cc:3058] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:58:50.815236 21122 raft_consensus.cc:3058] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:58:50.824479 21122 raft_consensus.cc:513] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:50.825115 21123 raft_consensus.cc:513] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:50.828135 21123 leader_election.cc:290] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: Requested vote from peers 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739), fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:50.831601 21058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" candidate_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "14f940533a744ceba9ebf397a1a45b8f"
I20250812 19:58:50.832365 21058 raft_consensus.cc:3058] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Advancing to term 3
W20250812 19:58:50.840849 20842 leader_election.cc:336] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
W20250812 19:58:50.841614 20842 leader_election.cc:336] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111)
I20250812 19:58:50.851011 21059 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c547a53cb7c647ad8ed0762e9470b5ee" candidate_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "14f940533a744ceba9ebf397a1a45b8f"
I20250812 19:58:50.851801 21059 raft_consensus.cc:3058] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:58:50.861227 21059 raft_consensus.cc:2466] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf17e5fbd8e148bfa3aa99f74803a8dc in term 3.
I20250812 19:58:50.861871 21122 leader_election.cc:290] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: Requested vote from peers fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977), 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739)
I20250812 19:58:50.863632 20841 leader_election.cc:304] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f940533a744ceba9ebf397a1a45b8f, cf17e5fbd8e148bfa3aa99f74803a8dc; no voters: fb685ac193094374b6df35b06da361e2
I20250812 19:58:50.865188 21122 raft_consensus.cc:2802] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 FOLLOWER]: Leader election won for term 3
I20250812 19:58:50.865857 21122 raft_consensus.cc:695] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 LEADER]: Becoming Leader. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Running, Role: LEADER
I20250812 19:58:50.866947 21122 consensus_queue.cc:237] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:50.871939 21058 raft_consensus.cc:2466] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf17e5fbd8e148bfa3aa99f74803a8dc in term 3.
I20250812 19:58:50.874411 20841 leader_election.cc:304] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f940533a744ceba9ebf397a1a45b8f, cf17e5fbd8e148bfa3aa99f74803a8dc; no voters: fb685ac193094374b6df35b06da361e2
I20250812 19:58:50.875458 21123 raft_consensus.cc:2802] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 FOLLOWER]: Leader election won for term 3
I20250812 19:58:50.894420 21123 raft_consensus.cc:695] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 LEADER]: Becoming Leader. State: Replica: cf17e5fbd8e148bfa3aa99f74803a8dc, State: Running, Role: LEADER
I20250812 19:58:50.895612 21123 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:50.896126 20766 catalog_manager.cc:5582] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: term changed from 2 to 3, leader changed from fb685ac193094374b6df35b06da361e2 (127.16.93.195) to cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193). New cstate: current_term: 3 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250812 19:58:50.936967 20766 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: term changed from 2 to 3, leader changed from <none> to cf17e5fbd8e148bfa3aa99f74803a8dc (127.16.93.193). New cstate: current_term: 3 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250812 19:58:51.280380 21058 raft_consensus.cc:1273] T c547a53cb7c647ad8ed0762e9470b5ee P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Refusing update from remote peer cf17e5fbd8e148bfa3aa99f74803a8dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250812 19:58:51.283038 21123 consensus_queue.cc:1035] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250812 19:58:51.322964 21059 raft_consensus.cc:1273] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Refusing update from remote peer cf17e5fbd8e148bfa3aa99f74803a8dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250812 19:58:51.325237 21123 consensus_queue.cc:1035] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250812 19:58:51.470178 20842 consensus_peers.cc:489] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc -> Peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Couldn't send request to peer fb685ac193094374b6df35b06da361e2. Status: Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250812 19:58:51.477604 20842 consensus_peers.cc:489] T c547a53cb7c647ad8ed0762e9470b5ee P cf17e5fbd8e148bfa3aa99f74803a8dc -> Peer fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): Couldn't send request to peer fb685ac193094374b6df35b06da361e2. Status: Network error: Client connection negotiation failed: client connection to 127.16.93.195:45977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250812 19:58:51.535099 20908 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:51.542490 21059 raft_consensus.cc:1273] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Refusing update from remote peer cf17e5fbd8e148bfa3aa99f74803a8dc: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250812 19:58:51.544375 21123 consensus_queue.cc:1035] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250812 19:58:51.552830 21128 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:51.559590 21057 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:51.583348 20753 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 21fd193e9c454edc98941ed66ec26d5a with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 19:58:51.589402 20766 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f reported cstate change: config changed from index 11 to 13, VOTER fb685ac193094374b6df35b06da361e2 (127.16.93.195) evicted. New cstate: current_term: 3 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
W20250812 19:58:51.640707 20766 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 21fd193e9c454edc98941ed66ec26d5a on TS fb685ac193094374b6df35b06da361e2: Not found: failed to reset TS proxy: Could not find TS for UUID fb685ac193094374b6df35b06da361e2
I20250812 19:58:51.647440 20908 consensus_queue.cc:237] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:51.655184 21128 raft_consensus.cc:2953] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } }
I20250812 19:58:51.670246 20753 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 21fd193e9c454edc98941ed66ec26d5a with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 19:58:51.675108 20765 catalog_manager.cc:5582] T 21fd193e9c454edc98941ed66ec26d5a P cf17e5fbd8e148bfa3aa99f74803a8dc reported cstate change: config changed from index 13 to 14, VOTER 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194) evicted. New cstate: current_term: 3 leader_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250812 19:58:51.693882 20751 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 21fd193e9c454edc98941ed66ec26d5a on TS fb685ac193094374b6df35b06da361e2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID fb685ac193094374b6df35b06da361e2
I20250812 19:58:51.727734 21039 tablet_service.cc:1515] Processing DeleteTablet for tablet 21fd193e9c454edc98941ed66ec26d5a with delete_type TABLET_DATA_TOMBSTONED (TS 14f940533a744ceba9ebf397a1a45b8f not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:51712
I20250812 19:58:51.743754 21148 tablet_replica.cc:331] stopping tablet replica
I20250812 19:58:51.744987 21148 raft_consensus.cc:2241] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250812 19:58:51.745867 21148 raft_consensus.cc:2270] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250812 19:58:51.750301 21148 ts_tablet_manager.cc:1905] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 19:58:51.783977 21148 ts_tablet_manager.cc:1918] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250812 19:58:51.797416 21148 log.cc:1199] T 21fd193e9c454edc98941ed66ec26d5a P 14f940533a744ceba9ebf397a1a45b8f: Deleting WAL directory at /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/wals/21fd193e9c454edc98941ed66ec26d5a
I20250812 19:58:51.800053 20752 catalog_manager.cc:4928] TS 14f940533a744ceba9ebf397a1a45b8f (127.16.93.194:41739): tablet 21fd193e9c454edc98941ed66ec26d5a (table TestTable [id=10cfd055e63e4937b0d9230df9d58691]) successfully deleted
W20250812 19:58:50.482656 21117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:52.158113 21108 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.676s user 0.508s sys 1.002s
W20250812 19:58:52.158476 21108 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.677s user 0.508s sys 1.002s
W20250812 19:58:51.884848 21115 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 21108
W20250812 19:58:52.161087 21119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:58:52.163232 21118 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1681 milliseconds
I20250812 19:58:52.163267 21108 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:58:52.164463 21108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:58:52.167063 21108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:58:52.168494 21108 hybrid_clock.cc:648] HybridClock initialized: now 1755028732168435 us; error 62 us; skew 500 ppm
I20250812 19:58:52.169600 21108 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:58:52.176815 21108 webserver.cc:480] Webserver started at http://127.16.93.195:40469/ using document root <none> and password file <none>
I20250812 19:58:52.178014 21108 fs_manager.cc:362] Metadata directory not provided
I20250812 19:58:52.178297 21108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:58:52.188491 21108 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250812 19:58:52.192973 21153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:58:52.194026 21108 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 19:58:52.194335 21108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb685ac193094374b6df35b06da361e2"
format_stamp: "Formatted at 2025-08-12 19:58:21 on dist-test-slave-wz96"
I20250812 19:58:52.196193 21108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:58:52.253132 21108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:58:52.254542 21108 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:58:52.254989 21108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:58:52.257457 21108 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:58:52.262916 21160 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250812 19:58:52.274494 21108 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250812 19:58:52.274752 21108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250812 19:58:52.275043 21108 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250812 19:58:52.280218 21160 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Bootstrap starting.
I20250812 19:58:52.282868 21108 ts_tablet_manager.cc:610] Registered 2 tablets
I20250812 19:58:52.283078 21108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250812 19:58:52.344106 21160 log.cc:826] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Log is configured to *not* fsync() on all Append() calls
I20250812 19:58:52.442116 21108 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:45977
I20250812 19:58:52.442173 21267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:45977 every 8 connection(s)
I20250812 19:58:52.445693 21108 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:58:52.455585 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21108
I20250812 19:58:52.473151 21160 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:52.474372 21160 tablet_bootstrap.cc:492] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Bootstrap complete.
I20250812 19:58:52.477068 21160 ts_tablet_manager.cc:1397] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Time spent bootstrapping tablet: real 0.197s user 0.163s sys 0.031s
I20250812 19:58:52.477782 21268 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35295
I20250812 19:58:52.478273 21268 heartbeater.cc:461] Registering TS with master...
I20250812 19:58:52.479506 21268 heartbeater.cc:507] Master 127.16.93.254:35295 requested a full tablet report, sending...
I20250812 19:58:52.484058 20765 ts_manager.cc:194] Registered new tserver with Master: fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977)
I20250812 19:58:52.487051 20765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:44753
I20250812 19:58:52.491374 21160 raft_consensus.cc:357] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:52.494055 21160 raft_consensus.cc:738] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb685ac193094374b6df35b06da361e2, State: Initialized, Role: FOLLOWER
I20250812 19:58:52.494853 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:58:52.494781 21160 consensus_queue.cc:260] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } attrs { promote: false } }
I20250812 19:58:52.497764 21268 heartbeater.cc:499] Master 127.16.93.254:35295 was elected leader, sending a full tablet report...
I20250812 19:58:52.498486 21160 ts_tablet_manager.cc:1428] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20250812 19:58:52.499186 21160 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap starting.
I20250812 19:58:52.499743 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250812 19:58:52.503459 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250812 19:58:52.583654 21160 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 19:58:52.584352 21160 tablet_bootstrap.cc:492] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Bootstrap complete.
I20250812 19:58:52.585517 21160 ts_tablet_manager.cc:1397] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Time spent bootstrapping tablet: real 0.086s user 0.077s sys 0.008s
I20250812 19:58:52.587143 21160 raft_consensus.cc:357] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:52.587574 21160 raft_consensus.cc:738] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb685ac193094374b6df35b06da361e2, State: Initialized, Role: FOLLOWER
I20250812 19:58:52.588110 21160 consensus_queue.cc:260] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fb685ac193094374b6df35b06da361e2" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 45977 } } peers { permanent_uuid: "14f940533a744ceba9ebf397a1a45b8f" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "cf17e5fbd8e148bfa3aa99f74803a8dc" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 41115 } attrs { promote: false } }
I20250812 19:58:52.589354 21160 ts_tablet_manager.cc:1428] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250812 19:58:52.714614 21223 raft_consensus.cc:3058] T c547a53cb7c647ad8ed0762e9470b5ee P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Advancing to term 3
I20250812 19:58:52.870446 21203 tablet_service.cc:1515] Processing DeleteTablet for tablet 21fd193e9c454edc98941ed66ec26d5a with delete_type TABLET_DATA_TOMBSTONED (TS fb685ac193094374b6df35b06da361e2 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:39072
I20250812 19:58:52.874830 21281 tablet_replica.cc:331] stopping tablet replica
I20250812 19:58:52.875450 21281 raft_consensus.cc:2241] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250812 19:58:52.875874 21281 raft_consensus.cc:2270] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 19:58:52.877866 21281 ts_tablet_manager.cc:1905] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 19:58:52.888624 21281 ts_tablet_manager.cc:1918] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250812 19:58:52.888962 21281 log.cc:1199] T 21fd193e9c454edc98941ed66ec26d5a P fb685ac193094374b6df35b06da361e2: Deleting WAL directory at /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/wals/21fd193e9c454edc98941ed66ec26d5a
I20250812 19:58:52.890148 20753 catalog_manager.cc:4928] TS fb685ac193094374b6df35b06da361e2 (127.16.93.195:45977): tablet 21fd193e9c454edc98941ed66ec26d5a (table TestTable [id=10cfd055e63e4937b0d9230df9d58691]) successfully deleted
W20250812 19:58:53.507462 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:58:54.512117 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:58:55.516113 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:58:56.519644 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:58:57.523029 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:58:58.526490 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:58:59.529964 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:00.533309 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:01.536688 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:02.539988 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:03.543484 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:04.547910 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:05.551698 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:06.555183 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:07.558485 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:08.561769 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:09.565128 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:10.568455 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 19:59:11.571792 16759 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 21fd193e9c454edc98941ed66ec26d5a: tablet_id: "21fd193e9c454edc98941ed66ec26d5a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250812 19:59:12.574900 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20803
I20250812 19:59:12.602845 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20957
I20250812 19:59:12.629630 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21108
I20250812 19:59:12.657052 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 20733
2025-08-12T19:59:12Z chronyd exiting
I20250812 19:59:12.705720 16759 test_util.cc:183] -----------------------------------------------
I20250812 19:59:12.705917 16759 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755028623597350-16759-0
[ FAILED ] AdminCliTest.TestRebuildTables (58913 ms)
[----------] 5 tests from AdminCliTest (129040 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250812 19:59:12.709717 16759 test_util.cc:276] Using random seed: 2036348988
I20250812 19:59:12.713922 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:59:12.714067 16759 ts_itest-base.cc:116] --------------
I20250812 19:59:12.714177 16759 ts_itest-base.cc:117] 5 tablet servers
I20250812 19:59:12.714284 16759 ts_itest-base.cc:118] 3 replicas per TS
I20250812 19:59:12.714385 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:59:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:59:12Z Disabled control of system clock
I20250812 19:59:12.757368 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35695
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:35695
--raft_prepare_replacement_before_eviction=true with env {}
W20250812 19:59:13.061757 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:13.062304 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:13.062716 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:13.093711 21303 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 19:59:13.094060 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:59:13.094266 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:13.094462 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:59:13.094664 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:59:13.131130 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:35695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:13.132438 21303 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:13.134063 21303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:13.146283 21309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:13.147200 21310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:13.148895 21312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:14.379508 21311 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1228 milliseconds
I20250812 19:59:14.379634 21303 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:14.380925 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:14.384044 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:14.385524 21303 hybrid_clock.cc:648] HybridClock initialized: now 1755028754385480 us; error 57 us; skew 500 ppm
I20250812 19:59:14.386380 21303 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:14.393081 21303 webserver.cc:480] Webserver started at http://127.16.93.254:41761/ using document root <none> and password file <none>
I20250812 19:59:14.394044 21303 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:14.394259 21303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:14.394752 21303 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:14.399286 21303 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6dd179f4e8584d41980313a2451d083b"
format_stamp: "Formatted at 2025-08-12 19:59:14 on dist-test-slave-wz96"
I20250812 19:59:14.400384 21303 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6dd179f4e8584d41980313a2451d083b"
format_stamp: "Formatted at 2025-08-12 19:59:14 on dist-test-slave-wz96"
I20250812 19:59:14.407459 21303 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250812 19:59:14.413285 21320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:14.414330 21303 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250812 19:59:14.414646 21303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6dd179f4e8584d41980313a2451d083b"
format_stamp: "Formatted at 2025-08-12 19:59:14 on dist-test-slave-wz96"
I20250812 19:59:14.414991 21303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:14.474838 21303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:14.476400 21303 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:14.476866 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:14.548333 21303 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:35695
I20250812 19:59:14.548439 21371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:35695 every 8 connection(s)
I20250812 19:59:14.551293 21303 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:59:14.557592 21372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:14.561702 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21303
I20250812 19:59:14.562296 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:59:14.579016 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b: Bootstrap starting.
I20250812 19:59:14.584785 21372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:14.586613 21372 log.cc:826] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:14.591274 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b: No bootstrap required, opened a new log
I20250812 19:59:14.609633 21372 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd179f4e8584d41980313a2451d083b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35695 } }
I20250812 19:59:14.610327 21372 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:14.610525 21372 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd179f4e8584d41980313a2451d083b, State: Initialized, Role: FOLLOWER
I20250812 19:59:14.611289 21372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd179f4e8584d41980313a2451d083b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35695 } }
I20250812 19:59:14.611935 21372 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:59:14.612308 21372 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:59:14.612672 21372 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:14.616900 21372 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd179f4e8584d41980313a2451d083b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35695 } }
I20250812 19:59:14.617640 21372 leader_election.cc:304] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dd179f4e8584d41980313a2451d083b; no voters:
I20250812 19:59:14.619367 21372 leader_election.cc:290] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:59:14.620162 21377 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:14.622505 21377 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [term 1 LEADER]: Becoming Leader. State: Replica: 6dd179f4e8584d41980313a2451d083b, State: Running, Role: LEADER
I20250812 19:59:14.623412 21377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd179f4e8584d41980313a2451d083b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35695 } }
I20250812 19:59:14.624500 21372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:59:14.635529 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dd179f4e8584d41980313a2451d083b. Latest consensus state: current_term: 1 leader_uuid: "6dd179f4e8584d41980313a2451d083b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd179f4e8584d41980313a2451d083b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35695 } } }
I20250812 19:59:14.635419 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dd179f4e8584d41980313a2451d083b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd179f4e8584d41980313a2451d083b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 35695 } } }
I20250812 19:59:14.636217 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:14.636438 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:14.639891 21384 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:59:14.652330 21384 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:59:14.669080 21384 catalog_manager.cc:1349] Generated new cluster ID: b8a152c183e74565a5a43a8a1d108818
I20250812 19:59:14.669400 21384 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:59:14.709563 21384 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:59:14.711318 21384 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:59:14.721720 21384 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6dd179f4e8584d41980313a2451d083b: Generated new TSK 0
I20250812 19:59:14.722806 21384 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:59:14.740765 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--builtin_ntp_servers=127.16.93.212:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250812 19:59:15.048501 21396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:15.049041 21396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:15.049619 21396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:15.080791 21396 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 19:59:15.081212 21396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:15.082001 21396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:59:15.118196 21396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:15.119491 21396 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:15.121093 21396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:15.133705 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:16.537377 21401 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 21396
W20250812 19:59:16.937261 21401 kernel_stack_watchdog.cc:198] Thread 21396 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:59:15.135329 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:16.938575 21404 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1803 milliseconds
W20250812 19:59:16.938805 21396 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.804s user 0.658s sys 1.080s
W20250812 19:59:16.939563 21396 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.805s user 0.658s sys 1.081s
I20250812 19:59:16.939901 21396 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 19:59:16.940043 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:16.943356 21396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:16.945719 21396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:16.947103 21396 hybrid_clock.cc:648] HybridClock initialized: now 1755028756947076 us; error 41 us; skew 500 ppm
I20250812 19:59:16.947893 21396 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:16.954118 21396 webserver.cc:480] Webserver started at http://127.16.93.193:45445/ using document root <none> and password file <none>
I20250812 19:59:16.955051 21396 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:16.955276 21396 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:16.955700 21396 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:16.960147 21396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3e37b8cb940c4543883cf948cb455372"
format_stamp: "Formatted at 2025-08-12 19:59:16 on dist-test-slave-wz96"
I20250812 19:59:16.961269 21396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3e37b8cb940c4543883cf948cb455372"
format_stamp: "Formatted at 2025-08-12 19:59:16 on dist-test-slave-wz96"
I20250812 19:59:16.968653 21396 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250812 19:59:16.974516 21412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:16.975766 21396 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 19:59:16.976070 21396 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3e37b8cb940c4543883cf948cb455372"
format_stamp: "Formatted at 2025-08-12 19:59:16 on dist-test-slave-wz96"
I20250812 19:59:16.976408 21396 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:17.032049 21396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:17.033597 21396 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:17.034034 21396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:17.036640 21396 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:17.040786 21396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:17.040992 21396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:17.041231 21396 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:17.041386 21396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:17.206880 21396 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:42251
I20250812 19:59:17.206992 21524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:42251 every 8 connection(s)
I20250812 19:59:17.209544 21396 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:59:17.218773 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21396
I20250812 19:59:17.219193 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:59:17.226265 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--builtin_ntp_servers=127.16.93.212:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 19:59:17.235972 21525 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35695
I20250812 19:59:17.236495 21525 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:17.237850 21525 heartbeater.cc:507] Master 127.16.93.254:35695 requested a full tablet report, sending...
I20250812 19:59:17.241075 21337 ts_manager.cc:194] Registered new tserver with Master: 3e37b8cb940c4543883cf948cb455372 (127.16.93.193:42251)
I20250812 19:59:17.243978 21337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:52379
W20250812 19:59:17.543784 21529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:17.544272 21529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:17.544821 21529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:17.589257 21529 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 19:59:17.589676 21529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:17.590461 21529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:59:17.625324 21529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:17.626670 21529 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:17.628229 21529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:17.639464 21535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:18.248138 21525 heartbeater.cc:499] Master 127.16.93.254:35695 was elected leader, sending a full tablet report...
W20250812 19:59:17.640987 21536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:18.834573 21538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:18.836428 21537 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1191 milliseconds
I20250812 19:59:18.836561 21529 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:18.837745 21529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:18.840323 21529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:18.841750 21529 hybrid_clock.cc:648] HybridClock initialized: now 1755028758841701 us; error 53 us; skew 500 ppm
I20250812 19:59:18.842509 21529 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:18.848876 21529 webserver.cc:480] Webserver started at http://127.16.93.194:36693/ using document root <none> and password file <none>
I20250812 19:59:18.849802 21529 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:18.849992 21529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:18.850487 21529 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:18.854791 21529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6a842013ac044820b6dbf738c9e3cfa9"
format_stamp: "Formatted at 2025-08-12 19:59:18 on dist-test-slave-wz96"
I20250812 19:59:18.855872 21529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6a842013ac044820b6dbf738c9e3cfa9"
format_stamp: "Formatted at 2025-08-12 19:59:18 on dist-test-slave-wz96"
I20250812 19:59:18.862707 21529 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250812 19:59:18.868117 21545 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:18.869082 21529 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250812 19:59:18.869385 21529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6a842013ac044820b6dbf738c9e3cfa9"
format_stamp: "Formatted at 2025-08-12 19:59:18 on dist-test-slave-wz96"
I20250812 19:59:18.869696 21529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:18.924182 21529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:18.925642 21529 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:18.926134 21529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:18.928622 21529 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:18.932556 21529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:18.932824 21529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250812 19:59:18.933101 21529 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:18.933243 21529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:19.071303 21529 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:40359
I20250812 19:59:19.071401 21657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:40359 every 8 connection(s)
I20250812 19:59:19.074086 21529 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 19:59:19.077129 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21529
I20250812 19:59:19.077831 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 19:59:19.085588 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--builtin_ntp_servers=127.16.93.212:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 19:59:19.099838 21658 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35695
I20250812 19:59:19.100373 21658 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:19.101740 21658 heartbeater.cc:507] Master 127.16.93.254:35695 requested a full tablet report, sending...
I20250812 19:59:19.104393 21337 ts_manager.cc:194] Registered new tserver with Master: 6a842013ac044820b6dbf738c9e3cfa9 (127.16.93.194:40359)
I20250812 19:59:19.105652 21337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:44703
W20250812 19:59:19.391471 21662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:19.392002 21662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:19.392477 21662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:19.423481 21662 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 19:59:19.423899 21662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:19.424721 21662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:59:19.464206 21662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:19.465703 21662 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:19.467434 21662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:19.480340 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:20.109686 21658 heartbeater.cc:499] Master 127.16.93.254:35695 was elected leader, sending a full tablet report...
W20250812 19:59:19.482084 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:20.685636 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:20.688445 21670 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1202 milliseconds
I20250812 19:59:20.688596 21662 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:20.689780 21662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:20.692451 21662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:20.693894 21662 hybrid_clock.cc:648] HybridClock initialized: now 1755028760693849 us; error 79 us; skew 500 ppm
I20250812 19:59:20.694797 21662 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:20.701344 21662 webserver.cc:480] Webserver started at http://127.16.93.195:38787/ using document root <none> and password file <none>
I20250812 19:59:20.702380 21662 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:20.702602 21662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:20.703027 21662 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:20.707368 21662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a7ff9361cef9462590f0600683cbeca4"
format_stamp: "Formatted at 2025-08-12 19:59:20 on dist-test-slave-wz96"
I20250812 19:59:20.708427 21662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a7ff9361cef9462590f0600683cbeca4"
format_stamp: "Formatted at 2025-08-12 19:59:20 on dist-test-slave-wz96"
I20250812 19:59:20.715440 21662 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250812 19:59:20.720883 21678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:20.721849 21662 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250812 19:59:20.722141 21662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a7ff9361cef9462590f0600683cbeca4"
format_stamp: "Formatted at 2025-08-12 19:59:20 on dist-test-slave-wz96"
I20250812 19:59:20.722445 21662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:20.785890 21662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:20.787364 21662 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:20.787796 21662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:20.790182 21662 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:20.794150 21662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:20.794348 21662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:20.794615 21662 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:20.794761 21662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:20.925136 21662 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:35965
I20250812 19:59:20.925243 21790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:35965 every 8 connection(s)
I20250812 19:59:20.927699 21662 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 19:59:20.934432 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21662
I20250812 19:59:20.934839 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 19:59:20.941010 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.196:0
--local_ip_for_outbound_sockets=127.16.93.196
--webserver_interface=127.16.93.196
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--builtin_ntp_servers=127.16.93.212:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 19:59:20.948020 21791 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35695
I20250812 19:59:20.948540 21791 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:20.949816 21791 heartbeater.cc:507] Master 127.16.93.254:35695 requested a full tablet report, sending...
I20250812 19:59:20.952100 21337 ts_manager.cc:194] Registered new tserver with Master: a7ff9361cef9462590f0600683cbeca4 (127.16.93.195:35965)
I20250812 19:59:20.953362 21337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:47233
W20250812 19:59:21.247602 21795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:21.248131 21795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:21.248651 21795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:21.281025 21795 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 19:59:21.281425 21795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:21.282228 21795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.196
I20250812 19:59:21.317159 21795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.93.196
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.196
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:21.318642 21795 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:21.320309 21795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:21.331318 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:21.956745 21791 heartbeater.cc:499] Master 127.16.93.254:35695 was elected leader, sending a full tablet report...
W20250812 19:59:21.336320 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:21.332902 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:22.503780 21803 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 19:59:22.503840 21795 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:22.507748 21795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:22.510646 21795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:22.512096 21795 hybrid_clock.cc:648] HybridClock initialized: now 1755028762512057 us; error 70 us; skew 500 ppm
I20250812 19:59:22.512952 21795 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:22.519330 21795 webserver.cc:480] Webserver started at http://127.16.93.196:35061/ using document root <none> and password file <none>
I20250812 19:59:22.520232 21795 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:22.520464 21795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:22.520961 21795 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:22.528182 21795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cdd4dc61978441f8b1c0f97f274e008a"
format_stamp: "Formatted at 2025-08-12 19:59:22 on dist-test-slave-wz96"
I20250812 19:59:22.529788 21795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cdd4dc61978441f8b1c0f97f274e008a"
format_stamp: "Formatted at 2025-08-12 19:59:22 on dist-test-slave-wz96"
I20250812 19:59:22.537954 21795 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250812 19:59:22.543480 21811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:22.544520 21795 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250812 19:59:22.544900 21795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cdd4dc61978441f8b1c0f97f274e008a"
format_stamp: "Formatted at 2025-08-12 19:59:22 on dist-test-slave-wz96"
I20250812 19:59:22.545217 21795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:22.610491 21795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:22.611950 21795 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:22.612394 21795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:22.614826 21795 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:22.618821 21795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:22.619005 21795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:22.619264 21795 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:22.619408 21795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:22.763561 21795 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.196:35675
I20250812 19:59:22.763748 21923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.196:35675 every 8 connection(s)
I20250812 19:59:22.766145 21795 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250812 19:59:22.767639 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21795
I20250812 19:59:22.768189 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250812 19:59:22.778986 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.197:0
--local_ip_for_outbound_sockets=127.16.93.197
--webserver_interface=127.16.93.197
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--builtin_ntp_servers=127.16.93.212:45973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 19:59:22.788022 21924 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35695
I20250812 19:59:22.788494 21924 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:22.789590 21924 heartbeater.cc:507] Master 127.16.93.254:35695 requested a full tablet report, sending...
I20250812 19:59:22.791987 21337 ts_manager.cc:194] Registered new tserver with Master: cdd4dc61978441f8b1c0f97f274e008a (127.16.93.196:35675)
I20250812 19:59:22.793918 21337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.196:39657
W20250812 19:59:23.098946 21928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:23.099438 21928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:23.099933 21928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:23.132442 21928 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 19:59:23.132905 21928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:23.133733 21928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.197
I20250812 19:59:23.168414 21928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:45973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.93.197
--webserver_port=0
--tserver_master_addrs=127.16.93.254:35695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.197
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:23.169907 21928 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:23.171551 21928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:23.184689 21935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:23.797928 21924 heartbeater.cc:499] Master 127.16.93.254:35695 was elected leader, sending a full tablet report...
W20250812 19:59:24.587744 21934 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 21928
W20250812 19:59:24.839351 21928 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.655s user 0.002s sys 0.000s
W20250812 19:59:24.839753 21928 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.655s user 0.002s sys 0.000s
W20250812 19:59:24.839897 21937 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1653 milliseconds
W20250812 19:59:23.185422 21936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:24.841189 21938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:24.841229 21928 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:24.844432 21928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:24.846490 21928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:24.847834 21928 hybrid_clock.cc:648] HybridClock initialized: now 1755028764847792 us; error 46 us; skew 500 ppm
I20250812 19:59:24.848621 21928 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:24.854439 21928 webserver.cc:480] Webserver started at http://127.16.93.197:40905/ using document root <none> and password file <none>
I20250812 19:59:24.855351 21928 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:24.855573 21928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:24.855990 21928 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:24.860316 21928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "aecad710916c4ab2b66e0949328b99c5"
format_stamp: "Formatted at 2025-08-12 19:59:24 on dist-test-slave-wz96"
I20250812 19:59:24.861446 21928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "aecad710916c4ab2b66e0949328b99c5"
format_stamp: "Formatted at 2025-08-12 19:59:24 on dist-test-slave-wz96"
I20250812 19:59:24.868412 21928 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250812 19:59:24.873827 21946 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:24.874867 21928 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250812 19:59:24.875202 21928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "aecad710916c4ab2b66e0949328b99c5"
format_stamp: "Formatted at 2025-08-12 19:59:24 on dist-test-slave-wz96"
I20250812 19:59:24.875588 21928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:24.930254 21928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:24.931649 21928 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:24.932070 21928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:24.934507 21928 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:24.938479 21928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:24.938688 21928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:24.938931 21928 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:24.939088 21928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:25.090689 21928 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.197:44407
I20250812 19:59:25.090795 22059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.197:44407 every 8 connection(s)
I20250812 19:59:25.093295 21928 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250812 19:59:25.100391 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 21928
I20250812 19:59:25.101529 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250812 19:59:25.115664 22060 heartbeater.cc:344] Connected to a master server at 127.16.93.254:35695
I20250812 19:59:25.116052 22060 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:25.117038 22060 heartbeater.cc:507] Master 127.16.93.254:35695 requested a full tablet report, sending...
I20250812 19:59:25.119109 21337 ts_manager.cc:194] Registered new tserver with Master: aecad710916c4ab2b66e0949328b99c5 (127.16.93.197:44407)
I20250812 19:59:25.120885 21337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.197:53035
I20250812 19:59:25.123540 16759 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250812 19:59:25.161021 21336 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250812 19:59:25.244289 21994 tablet_service.cc:1468] Processing CreateTablet for tablet 72c164a2a97c4386b88383668c7f67a6 (DEFAULT_TABLE table=TestTable [id=a014a1f7ec394f48a4fb1554620b26b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:25.246285 21859 tablet_service.cc:1468] Processing CreateTablet for tablet 72c164a2a97c4386b88383668c7f67a6 (DEFAULT_TABLE table=TestTable [id=a014a1f7ec394f48a4fb1554620b26b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:25.246373 21994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c164a2a97c4386b88383668c7f67a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:25.247725 21859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c164a2a97c4386b88383668c7f67a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:25.247820 21460 tablet_service.cc:1468] Processing CreateTablet for tablet 72c164a2a97c4386b88383668c7f67a6 (DEFAULT_TABLE table=TestTable [id=a014a1f7ec394f48a4fb1554620b26b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:25.249886 21460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c164a2a97c4386b88383668c7f67a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:25.272611 22079 tablet_bootstrap.cc:492] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a: Bootstrap starting.
I20250812 19:59:25.277611 22080 tablet_bootstrap.cc:492] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: Bootstrap starting.
I20250812 19:59:25.282541 22079 tablet_bootstrap.cc:654] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:25.285039 22081 tablet_bootstrap.cc:492] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: Bootstrap starting.
I20250812 19:59:25.285576 22079 log.cc:826] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:25.286890 22080 tablet_bootstrap.cc:654] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:25.289839 22080 log.cc:826] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:25.292150 22079 tablet_bootstrap.cc:492] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a: No bootstrap required, opened a new log
I20250812 19:59:25.292634 22079 ts_tablet_manager.cc:1397] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.002s
I20250812 19:59:25.293088 22081 tablet_bootstrap.cc:654] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:25.294989 22080 tablet_bootstrap.cc:492] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: No bootstrap required, opened a new log
I20250812 19:59:25.295153 22081 log.cc:826] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:25.295490 22080 ts_tablet_manager.cc:1397] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20250812 19:59:25.303994 22081 tablet_bootstrap.cc:492] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: No bootstrap required, opened a new log
I20250812 19:59:25.304498 22081 ts_tablet_manager.cc:1397] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20250812 19:59:25.321254 22079 raft_consensus.cc:357] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.322180 22079 raft_consensus.cc:383] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:25.321928 22080 raft_consensus.cc:357] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.322479 22079 raft_consensus.cc:738] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4dc61978441f8b1c0f97f274e008a, State: Initialized, Role: FOLLOWER
I20250812 19:59:25.322883 22080 raft_consensus.cc:383] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:25.323194 22080 raft_consensus.cc:738] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecad710916c4ab2b66e0949328b99c5, State: Initialized, Role: FOLLOWER
I20250812 19:59:25.323444 22079 consensus_queue.cc:260] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.324054 22080 consensus_queue.cc:260] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.325124 22081 raft_consensus.cc:357] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.326181 22081 raft_consensus.cc:383] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:25.326486 22081 raft_consensus.cc:738] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e37b8cb940c4543883cf948cb455372, State: Initialized, Role: FOLLOWER
I20250812 19:59:25.327829 22081 consensus_queue.cc:260] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.329275 22079 ts_tablet_manager.cc:1428] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a: Time spent starting tablet: real 0.036s user 0.019s sys 0.017s
I20250812 19:59:25.333441 22080 ts_tablet_manager.cc:1428] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: Time spent starting tablet: real 0.038s user 0.031s sys 0.009s
I20250812 19:59:25.333609 22081 ts_tablet_manager.cc:1428] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: Time spent starting tablet: real 0.029s user 0.023s sys 0.004s
I20250812 19:59:25.334365 22060 heartbeater.cc:499] Master 127.16.93.254:35695 was elected leader, sending a full tablet report...
W20250812 19:59:25.347607 22061 tablet.cc:2378] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:59:25.366451 22086 raft_consensus.cc:491] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:59:25.366896 22086 raft_consensus.cc:513] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.369204 22086 leader_election.cc:290] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4dc61978441f8b1c0f97f274e008a (127.16.93.196:35675), 3e37b8cb940c4543883cf948cb455372 (127.16.93.193:42251)
I20250812 19:59:25.377238 21879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72c164a2a97c4386b88383668c7f67a6" candidate_uuid: "aecad710916c4ab2b66e0949328b99c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4dc61978441f8b1c0f97f274e008a" is_pre_election: true
I20250812 19:59:25.378129 21879 raft_consensus.cc:2466] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecad710916c4ab2b66e0949328b99c5 in term 0.
I20250812 19:59:25.379462 21950 leader_election.cc:304] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aecad710916c4ab2b66e0949328b99c5, cdd4dc61978441f8b1c0f97f274e008a; no voters:
I20250812 19:59:25.380247 22086 raft_consensus.cc:2802] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:59:25.380522 22086 raft_consensus.cc:491] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:59:25.380795 22086 raft_consensus.cc:3058] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:25.381349 21480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72c164a2a97c4386b88383668c7f67a6" candidate_uuid: "aecad710916c4ab2b66e0949328b99c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e37b8cb940c4543883cf948cb455372" is_pre_election: true
I20250812 19:59:25.381989 21480 raft_consensus.cc:2466] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecad710916c4ab2b66e0949328b99c5 in term 0.
I20250812 19:59:25.385238 22086 raft_consensus.cc:513] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.387082 22086 leader_election.cc:290] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [CANDIDATE]: Term 1 election: Requested vote from peers cdd4dc61978441f8b1c0f97f274e008a (127.16.93.196:35675), 3e37b8cb940c4543883cf948cb455372 (127.16.93.193:42251)
I20250812 19:59:25.387696 21879 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72c164a2a97c4386b88383668c7f67a6" candidate_uuid: "aecad710916c4ab2b66e0949328b99c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4dc61978441f8b1c0f97f274e008a"
I20250812 19:59:25.387893 21480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72c164a2a97c4386b88383668c7f67a6" candidate_uuid: "aecad710916c4ab2b66e0949328b99c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e37b8cb940c4543883cf948cb455372"
I20250812 19:59:25.388118 21879 raft_consensus.cc:3058] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:25.388302 21480 raft_consensus.cc:3058] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:25.392786 21879 raft_consensus.cc:2466] T 72c164a2a97c4386b88383668c7f67a6 P cdd4dc61978441f8b1c0f97f274e008a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecad710916c4ab2b66e0949328b99c5 in term 1.
I20250812 19:59:25.392799 21480 raft_consensus.cc:2466] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecad710916c4ab2b66e0949328b99c5 in term 1.
I20250812 19:59:25.393566 21950 leader_election.cc:304] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e37b8cb940c4543883cf948cb455372, aecad710916c4ab2b66e0949328b99c5; no voters:
I20250812 19:59:25.394121 22086 raft_consensus.cc:2802] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:25.396085 22086 raft_consensus.cc:695] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [term 1 LEADER]: Becoming Leader. State: Replica: aecad710916c4ab2b66e0949328b99c5, State: Running, Role: LEADER
I20250812 19:59:25.396901 22086 consensus_queue.cc:237] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } }
I20250812 19:59:25.408329 21336 catalog_manager.cc:5582] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aecad710916c4ab2b66e0949328b99c5 (127.16.93.197). New cstate: current_term: 1 leader_uuid: "aecad710916c4ab2b66e0949328b99c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4dc61978441f8b1c0f97f274e008a" member_type: VOTER last_known_addr { host: "127.16.93.196" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aecad710916c4ab2b66e0949328b99c5" member_type: VOTER last_known_addr { host: "127.16.93.197" port: 44407 } health_report { overall_health: HEALTHY } } }
I20250812 19:59:25.444356 16759 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250812 19:59:25.447707 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3e37b8cb940c4543883cf948cb455372 to finish bootstrapping
I20250812 19:59:25.460925 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cdd4dc61978441f8b1c0f97f274e008a to finish bootstrapping
I20250812 19:59:25.471702 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver aecad710916c4ab2b66e0949328b99c5 to finish bootstrapping
W20250812 19:59:25.473140 21526 tablet.cc:2378] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:59:25.483630 16759 test_util.cc:276] Using random seed: 2049122899
I20250812 19:59:25.509634 16759 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250812 19:59:25.510569 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21795
I20250812 19:59:25.549661 21480 raft_consensus.cc:1273] T 72c164a2a97c4386b88383668c7f67a6 P 3e37b8cb940c4543883cf948cb455372 [term 1 FOLLOWER]: Refusing update from remote peer aecad710916c4ab2b66e0949328b99c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250812 19:59:25.551558 21950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.93.196:35675: connect: Connection refused (error 111)
I20250812 19:59:25.551352 22086 consensus_queue.cc:1035] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e37b8cb940c4543883cf948cb455372" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250812 19:59:25.560647 21950 consensus_peers.cc:489] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 -> Peer cdd4dc61978441f8b1c0f97f274e008a (127.16.93.196:35675): Couldn't send request to peer cdd4dc61978441f8b1c0f97f274e008a. Status: Network error: Client connection negotiation failed: client connection to 127.16.93.196:35675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250812 19:59:28.351786 21460 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 19:59:28.371767 21593 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 19:59:28.386940 21726 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 19:59:28.389415 21994 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250812 19:59:28.456202 21950 consensus_peers.cc:489] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 -> Peer cdd4dc61978441f8b1c0f97f274e008a (127.16.93.196:35675): Couldn't send request to peer cdd4dc61978441f8b1c0f97f274e008a. Status: Network error: Client connection negotiation failed: client connection to 127.16.93.196:35675: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250812 19:59:28.725776 21787 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac640 after lost signal to thread 21663
I20250812 19:59:30.284811 21460 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 19:59:30.367540 21593 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 19:59:30.396073 21994 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 19:59:30.424005 21726 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250812 19:59:30.929461 21950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.93.196:35675: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250812 19:59:30.932148 21950 consensus_peers.cc:489] T 72c164a2a97c4386b88383668c7f67a6 P aecad710916c4ab2b66e0949328b99c5 -> Peer cdd4dc61978441f8b1c0f97f274e008a (127.16.93.196:35675): Couldn't send request to peer cdd4dc61978441f8b1c0f97f274e008a. Status: Network error: Client connection negotiation failed: client connection to 127.16.93.196:35675: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250812 19:59:33.020231 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21396
I20250812 19:59:33.054055 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21529
I20250812 19:59:33.079977 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21662
I20250812 19:59:33.109809 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21928
I20250812 19:59:33.144642 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 21303
2025-08-12T19:59:33Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20493 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20493 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250812 19:59:33.203560 16759 test_util.cc:276] Using random seed: 2056842830
I20250812 19:59:33.207541 16759 ts_itest-base.cc:115] Starting cluster with:
I20250812 19:59:33.207720 16759 ts_itest-base.cc:116] --------------
I20250812 19:59:33.207895 16759 ts_itest-base.cc:117] 1 tablet servers
I20250812 19:59:33.208039 16759 ts_itest-base.cc:118] 1 replicas per TS
I20250812 19:59:33.208176 16759 ts_itest-base.cc:119] --------------
2025-08-12T19:59:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:59:33Z Disabled control of system clock
I20250812 19:59:33.252260 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:33325
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:33325 with env {}
W20250812 19:59:33.556594 22206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:33.557222 22206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:33.557680 22206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:33.588145 22206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:59:33.588456 22206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:33.588766 22206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:59:33.589038 22206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:59:33.624140 22206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:33325
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:33.625466 22206 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:33.627004 22206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:33.638249 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:33.638396 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:34.931720 22214 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1291 milliseconds
W20250812 19:59:34.932897 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:34.935778 22206 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.298s user 0.443s sys 0.840s
W20250812 19:59:34.936115 22206 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.298s user 0.445s sys 0.842s
I20250812 19:59:34.936307 22206 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:34.937427 22206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:34.939997 22206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:34.941390 22206 hybrid_clock.cc:648] HybridClock initialized: now 1755028774941357 us; error 45 us; skew 500 ppm
I20250812 19:59:34.942229 22206 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:34.949684 22206 webserver.cc:480] Webserver started at http://127.16.93.254:39763/ using document root <none> and password file <none>
I20250812 19:59:34.950750 22206 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:34.951001 22206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:34.951454 22206 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:34.955963 22206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "283d361977fd4bb89c15c4d30ded261b"
format_stamp: "Formatted at 2025-08-12 19:59:34 on dist-test-slave-wz96"
I20250812 19:59:34.957155 22206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "283d361977fd4bb89c15c4d30ded261b"
format_stamp: "Formatted at 2025-08-12 19:59:34 on dist-test-slave-wz96"
I20250812 19:59:34.965206 22206 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20250812 19:59:34.971407 22223 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:34.972671 22206 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250812 19:59:34.973016 22206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
uuid: "283d361977fd4bb89c15c4d30ded261b"
format_stamp: "Formatted at 2025-08-12 19:59:34 on dist-test-slave-wz96"
I20250812 19:59:34.973356 22206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:35.059163 22206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:35.060642 22206 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:35.061131 22206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:35.131585 22206 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:33325
I20250812 19:59:35.131646 22274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:33325 every 8 connection(s)
I20250812 19:59:35.134370 22206 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 19:59:35.139293 22275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:35.143525 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 22206
I20250812 19:59:35.144106 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 19:59:35.162220 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b: Bootstrap starting.
I20250812 19:59:35.167634 22275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:35.169883 22275 log.cc:826] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:35.174613 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b: No bootstrap required, opened a new log
I20250812 19:59:35.191285 22275 raft_consensus.cc:357] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d361977fd4bb89c15c4d30ded261b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33325 } }
I20250812 19:59:35.191946 22275 raft_consensus.cc:383] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:35.192171 22275 raft_consensus.cc:738] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283d361977fd4bb89c15c4d30ded261b, State: Initialized, Role: FOLLOWER
I20250812 19:59:35.192859 22275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d361977fd4bb89c15c4d30ded261b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33325 } }
I20250812 19:59:35.193349 22275 raft_consensus.cc:397] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:59:35.193653 22275 raft_consensus.cc:491] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:59:35.193959 22275 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:35.197962 22275 raft_consensus.cc:513] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d361977fd4bb89c15c4d30ded261b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33325 } }
I20250812 19:59:35.198823 22275 leader_election.cc:304] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283d361977fd4bb89c15c4d30ded261b; no voters:
I20250812 19:59:35.200538 22275 leader_election.cc:290] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:59:35.201188 22280 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:35.203445 22280 raft_consensus.cc:695] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [term 1 LEADER]: Becoming Leader. State: Replica: 283d361977fd4bb89c15c4d30ded261b, State: Running, Role: LEADER
I20250812 19:59:35.204196 22280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d361977fd4bb89c15c4d30ded261b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33325 } }
I20250812 19:59:35.205262 22275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:59:35.211541 22281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "283d361977fd4bb89c15c4d30ded261b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d361977fd4bb89c15c4d30ded261b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33325 } } }
I20250812 19:59:35.211696 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283d361977fd4bb89c15c4d30ded261b. Latest consensus state: current_term: 1 leader_uuid: "283d361977fd4bb89c15c4d30ded261b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d361977fd4bb89c15c4d30ded261b" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 33325 } } }
I20250812 19:59:35.212492 22281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:35.212666 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:35.216658 22287 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:59:35.230535 22287 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:59:35.245256 22287 catalog_manager.cc:1349] Generated new cluster ID: b6d5e464f64c471fae742aa6b5008de5
I20250812 19:59:35.245554 22287 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:59:35.265846 22287 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:59:35.267812 22287 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:59:35.283668 22287 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 283d361977fd4bb89c15c4d30ded261b: Generated new TSK 0
I20250812 19:59:35.284786 22287 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:59:35.303278 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33325
--builtin_ntp_servers=127.16.93.212:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 19:59:35.613529 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:35.614037 22299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:35.614530 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:35.646590 22299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:35.647536 22299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:59:35.682677 22299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:33325
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:35.683971 22299 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:35.685511 22299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:35.697475 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:35.698658 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:37.101286 22304 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 22299
W20250812 19:59:37.133963 22299 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.437s user 0.519s sys 0.919s
W20250812 19:59:37.135982 22299 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.440s user 0.519s sys 0.919s
I20250812 19:59:37.142561 22299 server_base.cc:1047] running on GCE node
W20250812 19:59:37.143347 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:37.144421 22299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:37.146441 22299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:37.147783 22299 hybrid_clock.cc:648] HybridClock initialized: now 1755028777147747 us; error 37 us; skew 500 ppm
I20250812 19:59:37.148531 22299 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:37.154856 22299 webserver.cc:480] Webserver started at http://127.16.93.193:44815/ using document root <none> and password file <none>
I20250812 19:59:37.155737 22299 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:37.155941 22299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:37.156366 22299 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:37.160431 22299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2853c80c090846938cdee720ccaa3597"
format_stamp: "Formatted at 2025-08-12 19:59:37 on dist-test-slave-wz96"
I20250812 19:59:37.161495 22299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2853c80c090846938cdee720ccaa3597"
format_stamp: "Formatted at 2025-08-12 19:59:37 on dist-test-slave-wz96"
I20250812 19:59:37.168888 22299 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250812 19:59:37.174798 22315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:37.175834 22299 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250812 19:59:37.176126 22299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2853c80c090846938cdee720ccaa3597"
format_stamp: "Formatted at 2025-08-12 19:59:37 on dist-test-slave-wz96"
I20250812 19:59:37.176450 22299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:37.233220 22299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:37.234841 22299 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:37.235283 22299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:37.238570 22299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:37.243868 22299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:37.244099 22299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:37.244381 22299 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:37.244554 22299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:37.413834 22299 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:38413
I20250812 19:59:37.413939 22427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:38413 every 8 connection(s)
I20250812 19:59:37.416492 22299 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 19:59:37.419008 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 22299
I20250812 19:59:37.419494 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755028623597350-16759-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 19:59:37.443300 22428 heartbeater.cc:344] Connected to a master server at 127.16.93.254:33325
I20250812 19:59:37.444092 22428 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:37.445566 22428 heartbeater.cc:507] Master 127.16.93.254:33325 requested a full tablet report, sending...
I20250812 19:59:37.448436 22240 ts_manager.cc:194] Registered new tserver with Master: 2853c80c090846938cdee720ccaa3597 (127.16.93.193:38413)
I20250812 19:59:37.450398 22240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:57235
I20250812 19:59:37.457798 16759 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250812 19:59:37.493657 22240 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36012:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250812 19:59:37.556428 22363 tablet_service.cc:1468] Processing CreateTablet for tablet 9ffeaf5478614d299d029fd7a2456033 (DEFAULT_TABLE table=TestTable [id=33ab95a85b1c4be89386c378920947da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:37.558003 22363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffeaf5478614d299d029fd7a2456033. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:37.578356 22443 tablet_bootstrap.cc:492] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597: Bootstrap starting.
I20250812 19:59:37.584236 22443 tablet_bootstrap.cc:654] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:37.586673 22443 log.cc:826] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:37.592658 22443 tablet_bootstrap.cc:492] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597: No bootstrap required, opened a new log
I20250812 19:59:37.593081 22443 ts_tablet_manager.cc:1397] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20250812 19:59:37.610121 22443 raft_consensus.cc:357] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2853c80c090846938cdee720ccaa3597" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38413 } }
I20250812 19:59:37.610746 22443 raft_consensus.cc:383] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:37.610982 22443 raft_consensus.cc:738] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2853c80c090846938cdee720ccaa3597, State: Initialized, Role: FOLLOWER
I20250812 19:59:37.611666 22443 consensus_queue.cc:260] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2853c80c090846938cdee720ccaa3597" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38413 } }
I20250812 19:59:37.612200 22443 raft_consensus.cc:397] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:59:37.612504 22443 raft_consensus.cc:491] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:59:37.612865 22443 raft_consensus.cc:3058] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:37.616936 22443 raft_consensus.cc:513] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2853c80c090846938cdee720ccaa3597" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38413 } }
I20250812 19:59:37.617607 22443 leader_election.cc:304] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2853c80c090846938cdee720ccaa3597; no voters:
I20250812 19:59:37.619803 22443 leader_election.cc:290] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:59:37.620193 22445 raft_consensus.cc:2802] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:37.623502 22428 heartbeater.cc:499] Master 127.16.93.254:33325 was elected leader, sending a full tablet report...
I20250812 19:59:37.624816 22443 ts_tablet_manager.cc:1428] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597: Time spent starting tablet: real 0.032s user 0.031s sys 0.001s
I20250812 19:59:37.625092 22445 raft_consensus.cc:695] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [term 1 LEADER]: Becoming Leader. State: Replica: 2853c80c090846938cdee720ccaa3597, State: Running, Role: LEADER
I20250812 19:59:37.625818 22445 consensus_queue.cc:237] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2853c80c090846938cdee720ccaa3597" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38413 } }
I20250812 19:59:37.637938 22240 catalog_manager.cc:5582] T 9ffeaf5478614d299d029fd7a2456033 P 2853c80c090846938cdee720ccaa3597 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2853c80c090846938cdee720ccaa3597 (127.16.93.193). New cstate: current_term: 1 leader_uuid: "2853c80c090846938cdee720ccaa3597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2853c80c090846938cdee720ccaa3597" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 38413 } health_report { overall_health: HEALTHY } } }
I20250812 19:59:37.659188 16759 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250812 19:59:37.662083 16759 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2853c80c090846938cdee720ccaa3597 to finish bootstrapping
I20250812 19:59:40.429620 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 22299
I20250812 19:59:40.454483 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 22206
2025-08-12T19:59:40Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7310 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7310 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250812 19:59:40.514917 16759 test_util.cc:276] Using random seed: 2064154186
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-12T19:59:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T19:59:40Z Disabled control of system clock
I20250812 19:59:40.566004 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44431
--webserver_interface=127.16.93.254
--webserver_port=0
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:44431 with env {}
W20250812 19:59:40.865507 22471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:40.866058 22471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:40.866463 22471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:40.897956 22471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:59:40.898249 22471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:40.898444 22471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:59:40.898628 22471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:59:40.934502 22471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:40.935897 22471 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:40.937469 22471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:40.947242 22477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:40.948833 22478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:42.135385 22480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:42.137614 22479 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1183 milliseconds
W20250812 19:59:42.138492 22471 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.190s user 0.006s sys 0.000s
W20250812 19:59:42.138774 22471 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.191s user 0.006s sys 0.000s
I20250812 19:59:42.139003 22471 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:42.140043 22471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:42.143080 22471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:42.144472 22471 hybrid_clock.cc:648] HybridClock initialized: now 1755028782144415 us; error 68 us; skew 500 ppm
I20250812 19:59:42.145305 22471 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:42.152940 22471 webserver.cc:480] Webserver started at http://127.16.93.254:43061/ using document root <none> and password file <none>
I20250812 19:59:42.153816 22471 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:42.154016 22471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:42.154443 22471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:42.158748 22471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/instance:
uuid: "4ffefd9ba34942f282c8da5d6ac6e28f"
format_stamp: "Formatted at 2025-08-12 19:59:42 on dist-test-slave-wz96"
I20250812 19:59:42.159775 22471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal/instance:
uuid: "4ffefd9ba34942f282c8da5d6ac6e28f"
format_stamp: "Formatted at 2025-08-12 19:59:42 on dist-test-slave-wz96"
I20250812 19:59:42.167311 22471 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250812 19:59:42.173025 22487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:42.174196 22471 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250812 19:59:42.174508 22471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
uuid: "4ffefd9ba34942f282c8da5d6ac6e28f"
format_stamp: "Formatted at 2025-08-12 19:59:42 on dist-test-slave-wz96"
I20250812 19:59:42.174835 22471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:42.243119 22471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:42.244617 22471 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:42.245052 22471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:42.316381 22471 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:44431
I20250812 19:59:42.316450 22538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:44431 every 8 connection(s)
I20250812 19:59:42.319098 22471 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb
I20250812 19:59:42.322227 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 22471
I20250812 19:59:42.322723 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal/instance
I20250812 19:59:42.324620 22539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:42.349788 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f: Bootstrap starting.
I20250812 19:59:42.355054 22539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:42.356835 22539 log.cc:826] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:42.361266 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f: No bootstrap required, opened a new log
I20250812 19:59:42.378562 22539 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 19:59:42.379222 22539 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:42.379468 22539 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffefd9ba34942f282c8da5d6ac6e28f, State: Initialized, Role: FOLLOWER
I20250812 19:59:42.380170 22539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 19:59:42.380679 22539 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:59:42.380957 22539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:59:42.381253 22539 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:42.385362 22539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 19:59:42.386242 22539 leader_election.cc:304] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ffefd9ba34942f282c8da5d6ac6e28f; no voters:
I20250812 19:59:42.388000 22539 leader_election.cc:290] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:59:42.388782 22544 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:42.390898 22544 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [term 1 LEADER]: Becoming Leader. State: Replica: 4ffefd9ba34942f282c8da5d6ac6e28f, State: Running, Role: LEADER
I20250812 19:59:42.391633 22544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 19:59:42.392616 22539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [sys.catalog]: configured and running, proceeding with master startup.
I20250812 19:59:42.398504 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } } }
I20250812 19:59:42.399219 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:42.399441 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ffefd9ba34942f282c8da5d6ac6e28f. Latest consensus state: current_term: 1 leader_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ffefd9ba34942f282c8da5d6ac6e28f" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } } }
I20250812 19:59:42.400056 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:42.405817 22550 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 19:59:42.417141 22550 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 19:59:42.433501 22550 catalog_manager.cc:1349] Generated new cluster ID: 5f1492f044d642f09fd5bdd7f9bf1887
I20250812 19:59:42.433779 22550 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 19:59:42.460400 22550 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 19:59:42.461879 22550 catalog_manager.cc:1506] Loading token signing keys...
I20250812 19:59:42.476869 22550 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4ffefd9ba34942f282c8da5d6ac6e28f: Generated new TSK 0
I20250812 19:59:42.477763 22550 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 19:59:42.485481 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:0
--local_ip_for_outbound_sockets=127.16.93.193
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44431
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250812 19:59:42.795024 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:42.795544 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:42.796036 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:42.827769 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:42.828670 22563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 19:59:42.864240 22563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:42.865578 22563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:42.867147 22563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:42.879971 22569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:42.880826 22570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:44.286479 22568 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 22563
W20250812 19:59:44.890743 22568 kernel_stack_watchdog.cc:198] Thread 22563 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:59:44.894646 22563 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.012s user 0.004s sys 0.000s
W20250812 19:59:44.895247 22563 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.013s user 0.005s sys 0.000s
W20250812 19:59:44.896765 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:44.897017 22563 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 19:59:44.897125 22571 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2013 milliseconds
I20250812 19:59:44.898622 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:44.901160 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:44.902647 22563 hybrid_clock.cc:648] HybridClock initialized: now 1755028784902596 us; error 45 us; skew 500 ppm
I20250812 19:59:44.903723 22563 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:44.911204 22563 webserver.cc:480] Webserver started at http://127.16.93.193:45253/ using document root <none> and password file <none>
I20250812 19:59:44.912461 22563 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:44.912752 22563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:44.913363 22563 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:44.919860 22563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/instance:
uuid: "60eda1e8d554494c8fddb4793ccbafda"
format_stamp: "Formatted at 2025-08-12 19:59:44 on dist-test-slave-wz96"
I20250812 19:59:44.921401 22563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal/instance:
uuid: "60eda1e8d554494c8fddb4793ccbafda"
format_stamp: "Formatted at 2025-08-12 19:59:44 on dist-test-slave-wz96"
I20250812 19:59:44.930823 22563 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.008s
I20250812 19:59:44.938246 22580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:44.939450 22563 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250812 19:59:44.939846 22563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
uuid: "60eda1e8d554494c8fddb4793ccbafda"
format_stamp: "Formatted at 2025-08-12 19:59:44 on dist-test-slave-wz96"
I20250812 19:59:44.940280 22563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:45.015236 22563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:45.016748 22563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:45.017182 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:45.020242 22563 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:45.024873 22563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:45.025089 22563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:45.025380 22563 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:45.025537 22563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:45.196486 22563 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:44421
I20250812 19:59:45.196684 22692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:44421 every 8 connection(s)
I20250812 19:59:45.199101 22563 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb
I20250812 19:59:45.205796 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 22563
I20250812 19:59:45.206590 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal/instance
I20250812 19:59:45.214973 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:0
--local_ip_for_outbound_sockets=127.16.93.194
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44431
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 19:59:45.225090 22693 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44431
I20250812 19:59:45.225541 22693 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:45.226574 22693 heartbeater.cc:507] Master 127.16.93.254:44431 requested a full tablet report, sending...
I20250812 19:59:45.229039 22504 ts_manager.cc:194] Registered new tserver with Master: 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421)
I20250812 19:59:45.231045 22504 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:58991
W20250812 19:59:45.581483 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:45.582075 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:45.582645 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:45.614254 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:45.615128 22697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 19:59:45.650300 22697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:45.651645 22697 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:45.653244 22697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:45.666081 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:46.235445 22693 heartbeater.cc:499] Master 127.16.93.254:44431 was elected leader, sending a full tablet report...
W20250812 19:59:45.668015 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:46.865182 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:46.867568 22705 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
I20250812 19:59:46.867671 22697 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:46.868906 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:46.872294 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:46.873646 22697 hybrid_clock.cc:648] HybridClock initialized: now 1755028786873608 us; error 39 us; skew 500 ppm
I20250812 19:59:46.874418 22697 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:46.880913 22697 webserver.cc:480] Webserver started at http://127.16.93.194:44947/ using document root <none> and password file <none>
I20250812 19:59:46.881892 22697 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:46.882164 22697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:46.882637 22697 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:46.887022 22697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/instance:
uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
format_stamp: "Formatted at 2025-08-12 19:59:46 on dist-test-slave-wz96"
I20250812 19:59:46.888130 22697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal/instance:
uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
format_stamp: "Formatted at 2025-08-12 19:59:46 on dist-test-slave-wz96"
I20250812 19:59:46.895141 22697 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250812 19:59:46.900679 22713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:46.901683 22697 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250812 19:59:46.902002 22697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
format_stamp: "Formatted at 2025-08-12 19:59:46 on dist-test-slave-wz96"
I20250812 19:59:46.902354 22697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:46.949607 22697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:46.951057 22697 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:46.951483 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:46.954124 22697 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:46.958351 22697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:46.958581 22697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:46.958786 22697 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:46.958918 22697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:47.089833 22697 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:32813
I20250812 19:59:47.089946 22825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:32813 every 8 connection(s)
I20250812 19:59:47.092353 22697 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb
I20250812 19:59:47.097352 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 22697
I20250812 19:59:47.097878 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal/instance
I20250812 19:59:47.104245 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:0
--local_ip_for_outbound_sockets=127.16.93.195
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44431
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 19:59:47.114058 22826 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44431
I20250812 19:59:47.114599 22826 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:47.115979 22826 heartbeater.cc:507] Master 127.16.93.254:44431 requested a full tablet report, sending...
I20250812 19:59:47.118606 22504 ts_manager.cc:194] Registered new tserver with Master: 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
I20250812 19:59:47.120431 22504 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:47961
W20250812 19:59:47.416484 22830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:47.417027 22830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:47.417464 22830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:47.451764 22830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:47.452617 22830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 19:59:47.489634 22830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=0
--tserver_master_addrs=127.16.93.254:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:47.490911 22830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:47.492419 22830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:47.503710 22836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:48.124717 22826 heartbeater.cc:499] Master 127.16.93.254:44431 was elected leader, sending a full tablet report...
W20250812 19:59:47.505108 22837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:48.702064 22839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:48.705495 22838 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1199 milliseconds
I20250812 19:59:48.705610 22830 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 19:59:48.706832 22830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 19:59:48.709506 22830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 19:59:48.710937 22830 hybrid_clock.cc:648] HybridClock initialized: now 1755028788710876 us; error 77 us; skew 500 ppm
I20250812 19:59:48.711872 22830 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:48.718590 22830 webserver.cc:480] Webserver started at http://127.16.93.195:36043/ using document root <none> and password file <none>
I20250812 19:59:48.719555 22830 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:48.719754 22830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:48.720242 22830 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:48.724900 22830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/instance:
uuid: "cbaa1e6b9d394f619027de86958dbc0b"
format_stamp: "Formatted at 2025-08-12 19:59:48 on dist-test-slave-wz96"
I20250812 19:59:48.726022 22830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal/instance:
uuid: "cbaa1e6b9d394f619027de86958dbc0b"
format_stamp: "Formatted at 2025-08-12 19:59:48 on dist-test-slave-wz96"
I20250812 19:59:48.733325 22830 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250812 19:59:48.739096 22846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:48.740180 22830 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250812 19:59:48.740514 22830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
uuid: "cbaa1e6b9d394f619027de86958dbc0b"
format_stamp: "Formatted at 2025-08-12 19:59:48 on dist-test-slave-wz96"
I20250812 19:59:48.740895 22830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:48.785586 22830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:48.787027 22830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:48.787479 22830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:48.789940 22830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 19:59:48.794064 22830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 19:59:48.794389 22830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:48.794672 22830 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 19:59:48.794833 22830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:48.928752 22830 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:42445
I20250812 19:59:48.928867 22958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:42445 every 8 connection(s)
I20250812 19:59:48.931605 22830 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb
I20250812 19:59:48.936072 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 22830
I20250812 19:59:48.936632 16759 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal/instance
I20250812 19:59:48.958366 22959 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44431
I20250812 19:59:48.958801 22959 heartbeater.cc:461] Registering TS with master...
I20250812 19:59:48.959798 22959 heartbeater.cc:507] Master 127.16.93.254:44431 requested a full tablet report, sending...
I20250812 19:59:48.961822 22504 ts_manager.cc:194] Registered new tserver with Master: cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445)
I20250812 19:59:48.963040 22504 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:59361
I20250812 19:59:48.972226 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 19:59:49.003440 16759 test_util.cc:276] Using random seed: 2072642708
I20250812 19:59:49.044566 22504 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42256:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250812 19:59:49.046923 22504 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 19:59:49.100430 22761 tablet_service.cc:1468] Processing CreateTablet for tablet a55ce92090654fa2a8d6f1b255dec9ee (DEFAULT_TABLE table=pre_rebuild [id=50464e0a381544d9a42009aaff535f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:49.102485 22761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a55ce92090654fa2a8d6f1b255dec9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:49.104029 22894 tablet_service.cc:1468] Processing CreateTablet for tablet a55ce92090654fa2a8d6f1b255dec9ee (DEFAULT_TABLE table=pre_rebuild [id=50464e0a381544d9a42009aaff535f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:49.107125 22894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a55ce92090654fa2a8d6f1b255dec9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:49.135582 22628 tablet_service.cc:1468] Processing CreateTablet for tablet a55ce92090654fa2a8d6f1b255dec9ee (DEFAULT_TABLE table=pre_rebuild [id=50464e0a381544d9a42009aaff535f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 19:59:49.138477 22628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a55ce92090654fa2a8d6f1b255dec9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:49.141086 22984 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Bootstrap starting.
I20250812 19:59:49.142278 22983 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Bootstrap starting.
I20250812 19:59:49.148926 22984 tablet_bootstrap.cc:654] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:49.150058 22983 tablet_bootstrap.cc:654] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:49.151358 22984 log.cc:826] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:49.152413 22983 log.cc:826] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:49.157212 22984 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: No bootstrap required, opened a new log
I20250812 19:59:49.157729 22984 ts_tablet_manager.cc:1397] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20250812 19:59:49.159771 22983 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: No bootstrap required, opened a new log
I20250812 19:59:49.160277 22983 ts_tablet_manager.cc:1397] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20250812 19:59:49.163527 22987 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Bootstrap starting.
I20250812 19:59:49.168941 22987 tablet_bootstrap.cc:654] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:49.170703 22987 log.cc:826] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:49.176688 22987 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: No bootstrap required, opened a new log
I20250812 19:59:49.177232 22987 ts_tablet_manager.cc:1397] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20250812 19:59:49.183976 22983 raft_consensus.cc:357] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.184837 22983 raft_consensus.cc:383] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:49.184280 22984 raft_consensus.cc:357] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.185119 22983 raft_consensus.cc:738] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f25919645fc4d6a8ca5540893d2a1d5, State: Initialized, Role: FOLLOWER
I20250812 19:59:49.185321 22984 raft_consensus.cc:383] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:49.185629 22984 raft_consensus.cc:738] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbaa1e6b9d394f619027de86958dbc0b, State: Initialized, Role: FOLLOWER
I20250812 19:59:49.186057 22983 consensus_queue.cc:260] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.186486 22984 consensus_queue.cc:260] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.190287 22983 ts_tablet_manager.cc:1428] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Time spent starting tablet: real 0.030s user 0.026s sys 0.003s
I20250812 19:59:49.192034 22959 heartbeater.cc:499] Master 127.16.93.254:44431 was elected leader, sending a full tablet report...
I20250812 19:59:49.192951 22984 ts_tablet_manager.cc:1428] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Time spent starting tablet: real 0.035s user 0.026s sys 0.010s
I20250812 19:59:49.206435 22987 raft_consensus.cc:357] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.207203 22987 raft_consensus.cc:383] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:49.207456 22987 raft_consensus.cc:738] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60eda1e8d554494c8fddb4793ccbafda, State: Initialized, Role: FOLLOWER
I20250812 19:59:49.208557 22987 consensus_queue.cc:260] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.212294 22987 ts_tablet_manager.cc:1428] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Time spent starting tablet: real 0.035s user 0.023s sys 0.008s
W20250812 19:59:49.349443 22827 tablet.cc:2378] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 19:59:49.436741 22960 tablet.cc:2378] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 19:59:49.458935 22694 tablet.cc:2378] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 19:59:49.521303 22991 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:59:49.521786 22991 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.524550 22990 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 19:59:49.525158 22990 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.525720 22991 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
I20250812 19:59:49.532205 22990 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813), 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421)
I20250812 19:59:49.548681 22914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbaa1e6b9d394f619027de86958dbc0b" is_pre_election: true
I20250812 19:59:49.549259 22781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" is_pre_election: true
I20250812 19:59:49.549558 22914 raft_consensus.cc:2466] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 0.
I20250812 19:59:49.550191 22781 raft_consensus.cc:2466] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 0.
I20250812 19:59:49.550745 22780 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "cbaa1e6b9d394f619027de86958dbc0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" is_pre_election: true
I20250812 19:59:49.551123 22582 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60eda1e8d554494c8fddb4793ccbafda, cbaa1e6b9d394f619027de86958dbc0b; no voters:
I20250812 19:59:49.551420 22780 raft_consensus.cc:2466] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbaa1e6b9d394f619027de86958dbc0b in term 0.
I20250812 19:59:49.552186 22991 raft_consensus.cc:2802] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:59:49.552542 22991 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:59:49.552917 22991 raft_consensus.cc:3058] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:49.553010 22849 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f25919645fc4d6a8ca5540893d2a1d5, cbaa1e6b9d394f619027de86958dbc0b; no voters:
I20250812 19:59:49.552976 22648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "cbaa1e6b9d394f619027de86958dbc0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60eda1e8d554494c8fddb4793ccbafda" is_pre_election: true
I20250812 19:59:49.553750 22990 raft_consensus.cc:2802] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 19:59:49.554040 22990 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 19:59:49.554302 22990 raft_consensus.cc:3058] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:49.558275 22991 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.559314 22648 raft_consensus.cc:2391] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cbaa1e6b9d394f619027de86958dbc0b in current term 1: Already voted for candidate 60eda1e8d554494c8fddb4793ccbafda in this term.
I20250812 19:59:49.559460 22990 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.560874 22914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbaa1e6b9d394f619027de86958dbc0b"
I20250812 19:59:49.561306 22990 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [CANDIDATE]: Term 1 election: Requested vote from peers 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813), 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421)
I20250812 19:59:49.561268 22780 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
I20250812 19:59:49.561676 22914 raft_consensus.cc:2391] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 60eda1e8d554494c8fddb4793ccbafda in current term 1: Already voted for candidate cbaa1e6b9d394f619027de86958dbc0b in this term.
I20250812 19:59:49.561813 22780 raft_consensus.cc:3058] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:49.562701 22991 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 election: Requested vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
I20250812 19:59:49.563021 22647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "cbaa1e6b9d394f619027de86958dbc0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60eda1e8d554494c8fddb4793ccbafda"
I20250812 19:59:49.564304 22781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "cbaa1e6b9d394f619027de86958dbc0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
I20250812 19:59:49.565958 22849 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbaa1e6b9d394f619027de86958dbc0b; no voters: 4f25919645fc4d6a8ca5540893d2a1d5, 60eda1e8d554494c8fddb4793ccbafda
I20250812 19:59:49.566689 22990 raft_consensus.cc:2747] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250812 19:59:49.567963 22780 raft_consensus.cc:2466] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 1.
I20250812 19:59:49.568881 22583 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f25919645fc4d6a8ca5540893d2a1d5, 60eda1e8d554494c8fddb4793ccbafda; no voters: cbaa1e6b9d394f619027de86958dbc0b
I20250812 19:59:49.569497 22991 raft_consensus.cc:2802] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:49.570969 22991 raft_consensus.cc:695] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 LEADER]: Becoming Leader. State: Replica: 60eda1e8d554494c8fddb4793ccbafda, State: Running, Role: LEADER
I20250812 19:59:49.571797 22991 consensus_queue.cc:237] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 19:59:49.580137 22502 catalog_manager.cc:5582] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda reported cstate change: term changed from 0 to 1, leader changed from <none> to 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193). New cstate: current_term: 1 leader_uuid: "60eda1e8d554494c8fddb4793ccbafda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } health_report { overall_health: HEALTHY } } }
I20250812 19:59:49.857849 22780 raft_consensus.cc:1273] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Refusing update from remote peer 60eda1e8d554494c8fddb4793ccbafda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 19:59:49.858057 22914 raft_consensus.cc:1273] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Refusing update from remote peer 60eda1e8d554494c8fddb4793ccbafda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 19:59:49.859606 22997 consensus_queue.cc:1035] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 19:59:49.860380 22991 consensus_queue.cc:1035] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 19:59:49.889616 23010 mvcc.cc:204] Tried to move back new op lower bound from 7188597923240742912 to 7188597922093207552. Current Snapshot: MvccSnapshot[applied={T|T < 7188597923240742912}]
I20250812 19:59:49.899443 23008 mvcc.cc:204] Tried to move back new op lower bound from 7188597923240742912 to 7188597922093207552. Current Snapshot: MvccSnapshot[applied={T|T < 7188597923240742912}]
I20250812 19:59:49.939870 23011 mvcc.cc:204] Tried to move back new op lower bound from 7188597923240742912 to 7188597922093207552. Current Snapshot: MvccSnapshot[applied={T|T < 7188597923240742912}]
W20250812 19:59:50.798707 22689 debug-util.cc:398] Leaking SignalData structure 0x7b08000acca0 after lost signal to thread 22564
I20250812 19:59:55.879192 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 22471
W20250812 19:59:55.954823 22959 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:44431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:44431: connect: Connection refused (error 111)
W20250812 19:59:56.067349 22826 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:44431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:44431: connect: Connection refused (error 111)
W20250812 19:59:56.070425 22693 heartbeater.cc:646] Failed to heartbeat to 127.16.93.254:44431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.93.254:44431: connect: Connection refused (error 111)
W20250812 19:59:56.255048 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:56.255681 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:56.287750 23047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 19:59:57.735208 23055 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 23047
W20250812 19:59:57.735769 23055 kernel_stack_watchdog.cc:198] Thread 23047 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 19:59:57.824982 23047 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.494s user 0.491s sys 0.948s
W20250812 19:59:57.924731 23047 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.594s user 0.493s sys 0.962s
I20250812 19:59:58.009676 23047 minidump.cc:252] Setting minidump size limit to 20M
I20250812 19:59:58.011533 23047 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:58.012571 23047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:58.022672 23081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:58.023471 23082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:58.025970 23084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 19:59:58.026149 23047 server_base.cc:1047] running on GCE node
I20250812 19:59:58.027227 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250812 19:59:58.027673 23047 hybrid_clock.cc:648] HybridClock initialized: now 1755028798027646 us; error 250516 us; skew 500 ppm
I20250812 19:59:58.028355 23047 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 19:59:58.032563 23047 webserver.cc:480] Webserver started at http://0.0.0.0:33589/ using document root <none> and password file <none>
I20250812 19:59:58.033442 23047 fs_manager.cc:362] Metadata directory not provided
I20250812 19:59:58.033667 23047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 19:59:58.034086 23047 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 19:59:58.038653 23047 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/instance:
uuid: "5f9a62973865426fa180a7767cf92324"
format_stamp: "Formatted at 2025-08-12 19:59:58 on dist-test-slave-wz96"
I20250812 19:59:58.039726 23047 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal/instance:
uuid: "5f9a62973865426fa180a7767cf92324"
format_stamp: "Formatted at 2025-08-12 19:59:58 on dist-test-slave-wz96"
I20250812 19:59:58.045410 23047 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250812 19:59:58.049892 23089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 19:59:58.050745 23047 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.003s
I20250812 19:59:58.051144 23047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
uuid: "5f9a62973865426fa180a7767cf92324"
format_stamp: "Formatted at 2025-08-12 19:59:58 on dist-test-slave-wz96"
I20250812 19:59:58.051470 23047 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 19:59:58.091289 23047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 19:59:58.093037 23047 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 19:59:58.093575 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 19:59:58.099689 23047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 19:59:58.121423 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Bootstrap starting.
I20250812 19:59:58.128082 23047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Neither blocks nor log segments found. Creating new log.
I20250812 19:59:58.130288 23047 log.cc:826] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Log is configured to *not* fsync() on all Append() calls
I20250812 19:59:58.135380 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: No bootstrap required, opened a new log
I20250812 19:59:58.152604 23047 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER }
I20250812 19:59:58.153199 23047 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 19:59:58.153411 23047 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f9a62973865426fa180a7767cf92324, State: Initialized, Role: FOLLOWER
I20250812 19:59:58.154104 23047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER }
I20250812 19:59:58.154526 23047 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 19:59:58.154728 23047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 19:59:58.155035 23047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 0 FOLLOWER]: Advancing to term 1
I20250812 19:59:58.158783 23047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER }
I20250812 19:59:58.159389 23047 leader_election.cc:304] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f9a62973865426fa180a7767cf92324; no voters:
I20250812 19:59:58.161111 23047 leader_election.cc:290] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 19:59:58.161398 23098 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 19:59:58.164036 23098 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 LEADER]: Becoming Leader. State: Replica: 5f9a62973865426fa180a7767cf92324, State: Running, Role: LEADER
I20250812 19:59:58.165009 23098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER }
I20250812 19:59:58.171577 23099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f9a62973865426fa180a7767cf92324. Latest consensus state: current_term: 1 leader_uuid: "5f9a62973865426fa180a7767cf92324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER } }
I20250812 19:59:58.172097 23099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:58.172942 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f9a62973865426fa180a7767cf92324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER } }
I20250812 19:59:58.173434 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: This master's current role is: LEADER
I20250812 19:59:58.184415 23047 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: stopping tablet replica
I20250812 19:59:58.185006 23047 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 LEADER]: Raft consensus shutting down.
I20250812 19:59:58.185400 23047 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250812 19:59:58.187644 23047 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250812 19:59:58.188062 23047 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250812 19:59:58.241618 23047 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250812 19:59:59.270596 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 22563
I20250812 19:59:59.308795 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 22697
I20250812 19:59:59.347038 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 22830
I20250812 19:59:59.383090 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44431
--webserver_interface=127.16.93.254
--webserver_port=43061
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.93.254:44431 with env {}
W20250812 19:59:59.691607 23107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 19:59:59.692229 23107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 19:59:59.692785 23107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 19:59:59.724051 23107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 19:59:59.724380 23107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 19:59:59.724684 23107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 19:59:59.724933 23107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 19:59:59.760250 23107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.93.254:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.93.254:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.93.254
--webserver_port=43061
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 19:59:59.761623 23107 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 19:59:59.763206 23107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 19:59:59.774021 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 19:59:59.774490 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 20:00:00.949738 23115 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250812 20:00:00.950984 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 20:00:00.951011 23107 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 20:00:00.954455 23107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 20:00:00.957540 23107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 20:00:00.958966 23107 hybrid_clock.cc:648] HybridClock initialized: now 1755028800958921 us; error 56 us; skew 500 ppm
I20250812 20:00:00.959762 23107 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 20:00:00.966361 23107 webserver.cc:480] Webserver started at http://127.16.93.254:43061/ using document root <none> and password file <none>
I20250812 20:00:00.967232 23107 fs_manager.cc:362] Metadata directory not provided
I20250812 20:00:00.967451 23107 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 20:00:00.974989 23107 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250812 20:00:00.979405 23123 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 20:00:00.980445 23107 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250812 20:00:00.980818 23107 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
uuid: "5f9a62973865426fa180a7767cf92324"
format_stamp: "Formatted at 2025-08-12 19:59:58 on dist-test-slave-wz96"
I20250812 20:00:00.982690 23107 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 20:00:01.028215 23107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 20:00:01.029691 23107 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 20:00:01.030117 23107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 20:00:01.099445 23107 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.254:44431
I20250812 20:00:01.099510 23174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.254:44431 every 8 connection(s)
I20250812 20:00:01.102360 23107 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb
I20250812 20:00:01.110611 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 23107
I20250812 20:00:01.112069 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.193:44421
--local_ip_for_outbound_sockets=127.16.93.193
--tserver_master_addrs=127.16.93.254:44431
--webserver_port=45253
--webserver_interface=127.16.93.193
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 20:00:01.112277 23175 sys_catalog.cc:263] Verifying existing consensus state
I20250812 20:00:01.124406 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Bootstrap starting.
I20250812 20:00:01.134186 23175 log.cc:826] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Log is configured to *not* fsync() on all Append() calls
I20250812 20:00:01.145756 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 20:00:01.146492 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Bootstrap complete.
I20250812 20:00:01.166442 23175 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 20:00:01.167064 23175 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f9a62973865426fa180a7767cf92324, State: Initialized, Role: FOLLOWER
I20250812 20:00:01.167800 23175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 20:00:01.168278 23175 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 20:00:01.168529 23175 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 20:00:01.168876 23175 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 1 FOLLOWER]: Advancing to term 2
I20250812 20:00:01.172808 23175 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 20:00:01.173461 23175 leader_election.cc:304] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f9a62973865426fa180a7767cf92324; no voters:
I20250812 20:00:01.175652 23175 leader_election.cc:290] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 20:00:01.176082 23179 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 20:00:01.178543 23179 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [term 2 LEADER]: Becoming Leader. State: Replica: 5f9a62973865426fa180a7767cf92324, State: Running, Role: LEADER
I20250812 20:00:01.179405 23179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } }
I20250812 20:00:01.181860 23175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 20:00:01.189337 23180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f9a62973865426fa180a7767cf92324. Latest consensus state: current_term: 2 leader_uuid: "5f9a62973865426fa180a7767cf92324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } } }
I20250812 20:00:01.190570 23180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: This master's current role is: LEADER
I20250812 20:00:01.193246 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5f9a62973865426fa180a7767cf92324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f9a62973865426fa180a7767cf92324" member_type: VOTER last_known_addr { host: "127.16.93.254" port: 44431 } } }
I20250812 20:00:01.194072 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324 [sys.catalog]: This master's current role is: LEADER
I20250812 20:00:01.194880 23185 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 20:00:01.208606 23185 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=a699787e432e4aacbe86ad8c02a2998b]
I20250812 20:00:01.218737 23185 tablet_loader.cc:96] loaded metadata for tablet a55ce92090654fa2a8d6f1b255dec9ee (table pre_rebuild [id=a699787e432e4aacbe86ad8c02a2998b])
I20250812 20:00:01.220657 23185 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 20:00:01.244168 23185 catalog_manager.cc:1349] Generated new cluster ID: e48e70bcf35c4e249d593a721f505d3c
I20250812 20:00:01.244479 23185 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 20:00:01.267246 23185 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 20:00:01.269098 23185 catalog_manager.cc:1506] Loading token signing keys...
I20250812 20:00:01.292990 23185 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Generated new TSK 0
I20250812 20:00:01.294018 23185 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 20:00:01.450207 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 20:00:01.450717 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 20:00:01.451212 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 20:00:01.483166 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 20:00:01.484040 23177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.193
I20250812 20:00:01.523406 23177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.193:44421
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.93.193
--webserver_port=45253
--tserver_master_addrs=127.16.93.254:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.193
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 20:00:01.524827 23177 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 20:00:01.526394 23177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 20:00:01.538841 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 20:00:01.540016 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 20:00:02.749832 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 20:00:02.751120 23177 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.210s user 0.005s sys 0.004s
W20250812 20:00:02.751524 23177 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.210s user 0.005s sys 0.004s
I20250812 20:00:02.751848 23177 server_base.cc:1047] running on GCE node
I20250812 20:00:02.753376 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 20:00:02.757103 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 20:00:02.758670 23177 hybrid_clock.cc:648] HybridClock initialized: now 1755028802758590 us; error 74 us; skew 500 ppm
I20250812 20:00:02.759850 23177 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 20:00:02.769224 23177 webserver.cc:480] Webserver started at http://127.16.93.193:45253/ using document root <none> and password file <none>
I20250812 20:00:02.770728 23177 fs_manager.cc:362] Metadata directory not provided
I20250812 20:00:02.771050 23177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 20:00:02.783439 23177 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.004s
I20250812 20:00:02.792515 23213 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 20:00:02.793912 23177 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20250812 20:00:02.794334 23177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
uuid: "60eda1e8d554494c8fddb4793ccbafda"
format_stamp: "Formatted at 2025-08-12 19:59:44 on dist-test-slave-wz96"
I20250812 20:00:02.797380 23177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 20:00:02.881572 23177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 20:00:02.883780 23177 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 20:00:02.884518 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 20:00:02.888419 23177 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 20:00:02.896639 23220 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 20:00:02.903903 23177 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 20:00:02.904120 23177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250812 20:00:02.904417 23177 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 20:00:02.908993 23177 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 20:00:02.909237 23177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250812 20:00:02.909610 23220 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Bootstrap starting.
I20250812 20:00:03.095419 23177 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.193:44421
I20250812 20:00:03.095664 23326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.193:44421 every 8 connection(s)
I20250812 20:00:03.098951 23177 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb
I20250812 20:00:03.101720 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 23177
I20250812 20:00:03.104108 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.194:32813
--local_ip_for_outbound_sockets=127.16.93.194
--tserver_master_addrs=127.16.93.254:44431
--webserver_port=44947
--webserver_interface=127.16.93.194
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 20:00:03.211885 23327 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44431
I20250812 20:00:03.212553 23327 heartbeater.cc:461] Registering TS with master...
I20250812 20:00:03.213950 23327 heartbeater.cc:507] Master 127.16.93.254:44431 requested a full tablet report, sending...
I20250812 20:00:03.218858 23140 ts_manager.cc:194] Registered new tserver with Master: 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421)
I20250812 20:00:03.226274 23140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.193:59081
I20250812 20:00:03.237089 23220 log.cc:826] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Log is configured to *not* fsync() on all Append() calls
W20250812 20:00:03.502607 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 20:00:03.503108 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 20:00:03.503603 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 20:00:03.534773 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 20:00:03.535674 23331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.194
I20250812 20:00:03.570726 23331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.194:32813
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.93.194
--webserver_port=44947
--tserver_master_addrs=127.16.93.254:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.194
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 20:00:03.572052 23331 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 20:00:03.573976 23331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 20:00:03.587265 23338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 20:00:04.230873 23327 heartbeater.cc:499] Master 127.16.93.254:44431 was elected leader, sending a full tablet report...
W20250812 20:00:03.598980 23339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 20:00:04.794045 23340 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1204 milliseconds
I20250812 20:00:04.794144 23331 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 20:00:03.597425 23341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 20:00:04.799489 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 20:00:04.802217 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 20:00:04.803691 23331 hybrid_clock.cc:648] HybridClock initialized: now 1755028804803641 us; error 54 us; skew 500 ppm
I20250812 20:00:04.804737 23331 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 20:00:04.812098 23331 webserver.cc:480] Webserver started at http://127.16.93.194:44947/ using document root <none> and password file <none>
I20250812 20:00:04.813342 23331 fs_manager.cc:362] Metadata directory not provided
I20250812 20:00:04.813629 23331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 20:00:04.823971 23331 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.005s
I20250812 20:00:04.829771 23349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 20:00:04.830955 23331 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250812 20:00:04.831328 23331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
format_stamp: "Formatted at 2025-08-12 19:59:46 on dist-test-slave-wz96"
I20250812 20:00:04.834061 23331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 20:00:04.911497 23331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 20:00:04.912987 23331 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 20:00:04.913403 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 20:00:04.916074 23331 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 20:00:04.922024 23356 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 20:00:04.932641 23331 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 20:00:04.932916 23331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250812 20:00:04.933192 23331 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 20:00:04.937947 23331 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 20:00:04.938150 23331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.004s
I20250812 20:00:04.938601 23356 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Bootstrap starting.
I20250812 20:00:05.199717 23331 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.194:32813
I20250812 20:00:05.199832 23462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.194:32813 every 8 connection(s)
I20250812 20:00:05.204113 23331 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb
I20250812 20:00:05.214725 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 23331
I20250812 20:00:05.220955 16759 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
/tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.93.195:42445
--local_ip_for_outbound_sockets=127.16.93.195
--tserver_master_addrs=127.16.93.254:44431
--webserver_port=36043
--webserver_interface=127.16.93.195
--builtin_ntp_servers=127.16.93.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 20:00:05.241839 23356 log.cc:826] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Log is configured to *not* fsync() on all Append() calls
I20250812 20:00:05.247900 23463 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44431
I20250812 20:00:05.248392 23463 heartbeater.cc:461] Registering TS with master...
I20250812 20:00:05.249709 23463 heartbeater.cc:507] Master 127.16.93.254:44431 requested a full tablet report, sending...
I20250812 20:00:05.253587 23140 ts_manager.cc:194] Registered new tserver with Master: 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
I20250812 20:00:05.256516 23140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.194:43677
I20250812 20:00:05.656822 23220 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 20:00:05.657613 23220 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Bootstrap complete.
I20250812 20:00:05.658952 23220 ts_tablet_manager.cc:1397] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Time spent bootstrapping tablet: real 2.750s user 2.640s sys 0.072s
I20250812 20:00:05.669893 23220 raft_consensus.cc:357] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:05.671985 23220 raft_consensus.cc:738] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60eda1e8d554494c8fddb4793ccbafda, State: Initialized, Role: FOLLOWER
I20250812 20:00:05.672832 23220 consensus_queue.cc:260] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:05.676282 23220 ts_tablet_manager.cc:1428] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20250812 20:00:05.684746 23467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 20:00:05.685369 23467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 20:00:05.686126 23467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 20:00:05.741381 23467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 20:00:05.742784 23467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.93.195
I20250812 20:00:05.803246 23467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.93.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.93.195:42445
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.93.195
--webserver_port=36043
--tserver_master_addrs=127.16.93.254:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.93.195
--log_dir=/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7537
TSAN enabled
I20250812 20:00:05.804834 23467 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 20:00:05.806366 23467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 20:00:05.819360 23475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 20:00:06.260555 23463 heartbeater.cc:499] Master 127.16.93.254:44431 was elected leader, sending a full tablet report...
W20250812 20:00:05.836463 23476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 20:00:05.840410 23478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 20:00:05.837997 23467 server_base.cc:1047] running on GCE node
I20250812 20:00:07.032122 23467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 20:00:07.034942 23467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 20:00:07.036468 23467 hybrid_clock.cc:648] HybridClock initialized: now 1755028807036411 us; error 53 us; skew 500 ppm
I20250812 20:00:07.037536 23467 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 20:00:07.045030 23467 webserver.cc:480] Webserver started at http://127.16.93.195:36043/ using document root <none> and password file <none>
I20250812 20:00:07.046293 23467 fs_manager.cc:362] Metadata directory not provided
I20250812 20:00:07.046574 23467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 20:00:07.057250 23467 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250812 20:00:07.063072 23485 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 20:00:07.064431 23467 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 20:00:07.064708 23486 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 20:00:07.065244 23486 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:07.064815 23467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
uuid: "cbaa1e6b9d394f619027de86958dbc0b"
format_stamp: "Formatted at 2025-08-12 19:59:48 on dist-test-slave-wz96"
I20250812 20:00:07.066838 23467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 20:00:07.075253 23486 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
W20250812 20:00:07.084537 23215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.93.195:42445: connect: Connection refused (error 111)
W20250812 20:00:07.112843 23215 leader_election.cc:336] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445): Network error: Client connection negotiation failed: client connection to 127.16.93.195:42445: connect: Connection refused (error 111)
I20250812 20:00:07.098560 23418 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" is_pre_election: true
W20250812 20:00:07.118643 23216 leader_election.cc:343] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813): Illegal state: must be running to vote when last-logged opid is not known
I20250812 20:00:07.119146 23216 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60eda1e8d554494c8fddb4793ccbafda; no voters: 4f25919645fc4d6a8ca5540893d2a1d5, cbaa1e6b9d394f619027de86958dbc0b
I20250812 20:00:07.120368 23486 raft_consensus.cc:2747] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250812 20:00:07.152383 23467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 20:00:07.153863 23467 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 20:00:07.154311 23467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 20:00:07.157009 23467 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 20:00:07.163319 23497 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 20:00:07.173923 23467 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 20:00:07.174177 23467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250812 20:00:07.174450 23467 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 20:00:07.179263 23467 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 20:00:07.179457 23467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250812 20:00:07.179916 23497 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Bootstrap starting.
I20250812 20:00:07.446703 23467 rpc_server.cc:307] RPC server started. Bound to: 127.16.93.195:42445
I20250812 20:00:07.446902 23603 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.93.195:42445 every 8 connection(s)
I20250812 20:00:07.450533 23467 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb
I20250812 20:00:07.459100 16759 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu as pid 23467
I20250812 20:00:07.487926 23604 heartbeater.cc:344] Connected to a master server at 127.16.93.254:44431
I20250812 20:00:07.488711 23604 heartbeater.cc:461] Registering TS with master...
I20250812 20:00:07.490263 23604 heartbeater.cc:507] Master 127.16.93.254:44431 requested a full tablet report, sending...
I20250812 20:00:07.494307 23140 ts_manager.cc:194] Registered new tserver with Master: cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445)
I20250812 20:00:07.497349 23140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.93.195:34379
I20250812 20:00:07.503805 16759 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 20:00:07.507984 23497 log.cc:826] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Log is configured to *not* fsync() on all Append() calls
I20250812 20:00:07.543367 23356 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 20:00:07.544143 23356 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Bootstrap complete.
I20250812 20:00:07.545665 23356 ts_tablet_manager.cc:1397] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Time spent bootstrapping tablet: real 2.607s user 2.530s sys 0.062s
I20250812 20:00:07.550593 23356 raft_consensus.cc:357] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:07.552565 23356 raft_consensus.cc:738] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f25919645fc4d6a8ca5540893d2a1d5, State: Initialized, Role: FOLLOWER
I20250812 20:00:07.553283 23356 consensus_queue.cc:260] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:07.556349 23356 ts_tablet_manager.cc:1428] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20250812 20:00:08.501439 23604 heartbeater.cc:499] Master 127.16.93.254:44431 was elected leader, sending a full tablet report...
I20250812 20:00:09.145466 23617 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 20:00:09.145932 23617 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:09.154413 23617 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421)
I20250812 20:00:09.175338 23282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "60eda1e8d554494c8fddb4793ccbafda" is_pre_election: true
I20250812 20:00:09.176155 23282 raft_consensus.cc:2466] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f25919645fc4d6a8ca5540893d2a1d5 in term 1.
I20250812 20:00:09.177829 23353 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f25919645fc4d6a8ca5540893d2a1d5, 60eda1e8d554494c8fddb4793ccbafda; no voters:
I20250812 20:00:09.178723 23617 raft_consensus.cc:2802] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250812 20:00:09.179090 23617 raft_consensus.cc:491] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 20:00:09.179420 23617 raft_consensus.cc:3058] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Advancing to term 2
I20250812 20:00:09.173439 23559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "cbaa1e6b9d394f619027de86958dbc0b" is_pre_election: true
W20250812 20:00:09.184248 23351 leader_election.cc:343] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445): Illegal state: must be running to vote when last-logged opid is not known
I20250812 20:00:09.188474 23617 raft_consensus.cc:513] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:09.190373 23617 leader_election.cc:290] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [CANDIDATE]: Term 2 election: Requested vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421)
I20250812 20:00:09.191239 23559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "cbaa1e6b9d394f619027de86958dbc0b"
I20250812 20:00:09.191460 23282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a55ce92090654fa2a8d6f1b255dec9ee" candidate_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "60eda1e8d554494c8fddb4793ccbafda"
I20250812 20:00:09.192000 23282 raft_consensus.cc:3058] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Advancing to term 2
W20250812 20:00:09.192188 23351 leader_election.cc:343] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445): Illegal state: must be running to vote when last-logged opid is not known
I20250812 20:00:09.201208 23282 raft_consensus.cc:2466] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f25919645fc4d6a8ca5540893d2a1d5 in term 2.
I20250812 20:00:09.202322 23353 leader_election.cc:304] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f25919645fc4d6a8ca5540893d2a1d5, 60eda1e8d554494c8fddb4793ccbafda; no voters: cbaa1e6b9d394f619027de86958dbc0b
I20250812 20:00:09.203157 23617 raft_consensus.cc:2802] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 20:00:09.204773 23617 raft_consensus.cc:695] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 LEADER]: Becoming Leader. State: Replica: 4f25919645fc4d6a8ca5540893d2a1d5, State: Running, Role: LEADER
I20250812 20:00:09.205771 23617 consensus_queue.cc:237] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:09.218890 23140 catalog_manager.cc:5582] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 reported cstate change: term changed from 0 to 2, leader changed from <none> to 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194), VOTER 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194) added, VOTER 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193) added, VOTER cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195) added. New cstate: current_term: 2 leader_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } health_report { overall_health: UNKNOWN } } }
I20250812 20:00:09.433917 23497 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 20:00:09.434741 23497 tablet_bootstrap.cc:492] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Bootstrap complete.
I20250812 20:00:09.436079 23497 ts_tablet_manager.cc:1397] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Time spent bootstrapping tablet: real 2.257s user 2.175s sys 0.060s
I20250812 20:00:09.444639 23497 raft_consensus.cc:357] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:09.447659 23497 raft_consensus.cc:738] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbaa1e6b9d394f619027de86958dbc0b, State: Initialized, Role: FOLLOWER
I20250812 20:00:09.448388 23497 consensus_queue.cc:260] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:09.451375 23497 ts_tablet_manager.cc:1428] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20250812 20:00:09.732204 23282 raft_consensus.cc:1273] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 2 FOLLOWER]: Refusing update from remote peer 4f25919645fc4d6a8ca5540893d2a1d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250812 20:00:09.733886 23634 consensus_queue.cc:1035] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250812 20:00:09.761833 23559 raft_consensus.cc:3058] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Advancing to term 2
I20250812 20:00:09.772233 23559 raft_consensus.cc:1273] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 2 FOLLOWER]: Refusing update from remote peer 4f25919645fc4d6a8ca5540893d2a1d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250812 20:00:09.780053 23637 consensus_queue.cc:1035] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20250812 20:00:09.826352 23418 consensus_queue.cc:237] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } }
I20250812 20:00:09.832135 23282 raft_consensus.cc:1273] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 2 FOLLOWER]: Refusing update from remote peer 4f25919645fc4d6a8ca5540893d2a1d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250812 20:00:09.833427 23637 consensus_queue.cc:1035] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250812 20:00:09.838191 23634 raft_consensus.cc:2953] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } }
I20250812 20:00:09.839911 23282 raft_consensus.cc:2953] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } }
I20250812 20:00:09.851161 23126 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a55ce92090654fa2a8d6f1b255dec9ee with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 20:00:09.852100 23140 catalog_manager.cc:5582] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda reported cstate change: config changed from index -1 to 207, VOTER cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195) evicted. New cstate: current_term: 2 leader_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } }
I20250812 20:00:09.871451 23418 consensus_queue.cc:237] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:09.879019 23634 raft_consensus.cc:2953] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } } }
I20250812 20:00:09.894749 23126 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a55ce92090654fa2a8d6f1b255dec9ee with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 20:00:09.900238 23139 catalog_manager.cc:5582] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 reported cstate change: config changed from index 207 to 208, VOTER 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193) evicted. New cstate: current_term: 2 leader_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } health_report { overall_health: HEALTHY } } }
I20250812 20:00:09.915556 23539 tablet_service.cc:1515] Processing DeleteTablet for tablet a55ce92090654fa2a8d6f1b255dec9ee with delete_type TABLET_DATA_TOMBSTONED (TS cbaa1e6b9d394f619027de86958dbc0b not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:52924
I20250812 20:00:09.926067 23650 tablet_replica.cc:331] stopping tablet replica
I20250812 20:00:09.926927 23650 raft_consensus.cc:2241] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250812 20:00:09.928148 23650 raft_consensus.cc:2270] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 20:00:09.928468 23262 tablet_service.cc:1515] Processing DeleteTablet for tablet a55ce92090654fa2a8d6f1b255dec9ee with delete_type TABLET_DATA_TOMBSTONED (TS 60eda1e8d554494c8fddb4793ccbafda not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:47328
I20250812 20:00:09.942549 23651 tablet_replica.cc:331] stopping tablet replica
I20250812 20:00:09.943432 23651 raft_consensus.cc:2241] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 2 FOLLOWER]: Raft consensus shutting down.
I20250812 20:00:09.944103 23651 raft_consensus.cc:2270] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 20:00:09.958349 23650 ts_tablet_manager.cc:1905] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 20:00:09.970793 23651 ts_tablet_manager.cc:1905] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 20:00:09.973783 23650 ts_tablet_manager.cc:1918] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250812 20:00:09.974217 23650 log.cc:1199] T a55ce92090654fa2a8d6f1b255dec9ee P cbaa1e6b9d394f619027de86958dbc0b: Deleting WAL directory at /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/wal/wals/a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:09.976214 23125 catalog_manager.cc:4928] TS cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445): tablet a55ce92090654fa2a8d6f1b255dec9ee (table pre_rebuild [id=a699787e432e4aacbe86ad8c02a2998b]) successfully deleted
I20250812 20:00:09.983702 23651 ts_tablet_manager.cc:1918] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250812 20:00:09.984026 23651 log.cc:1199] T a55ce92090654fa2a8d6f1b255dec9ee P 60eda1e8d554494c8fddb4793ccbafda: Deleting WAL directory at /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/wal/wals/a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:09.985679 23127 catalog_manager.cc:4928] TS 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193:44421): tablet a55ce92090654fa2a8d6f1b255dec9ee (table pre_rebuild [id=a699787e432e4aacbe86ad8c02a2998b]) successfully deleted
W20250812 20:00:10.010547 16759 scanner-internal.cc:458] Time spent opening tablet: real 2.468s user 0.007s sys 0.002s
I20250812 20:00:10.734026 23262 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 20:00:10.739599 23398 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 20:00:10.749617 23539 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5f9a62973865426fa180a7767cf92324 | 127.16.93.254:44431 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.93.212:33599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
4f25919645fc4d6a8ca5540893d2a1d5 | 127.16.93.194:32813 | HEALTHY | <none> | 1 | 0
60eda1e8d554494c8fddb4793ccbafda | 127.16.93.193:44421 | HEALTHY | <none> | 0 | 0
cbaa1e6b9d394f619027de86958dbc0b | 127.16.93.195:42445 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.16.93.193 | experimental | 127.16.93.193:44421
local_ip_for_outbound_sockets | 127.16.93.194 | experimental | 127.16.93.194:32813
local_ip_for_outbound_sockets | 127.16.93.195 | experimental | 127.16.93.195:42445
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.93.193:44421
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.93.194:32813
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb | hidden | 127.16.93.195:42445
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.93.212:33599 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250812 20:00:11.002713 16759 log_verifier.cc:126] Checking tablet a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:11.302706 16759 log_verifier.cc:177] Verified matching terms for 208 ops in tablet a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:11.305298 23140 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52528:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250812 20:00:11.305887 23140 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52528:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250812 20:00:11.317938 23140 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f9a62973865426fa180a7767cf92324: Sending DeleteTablet for 1 replicas of tablet a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:11.319707 23398 tablet_service.cc:1515] Processing DeleteTablet for tablet a55ce92090654fa2a8d6f1b255dec9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-12 20:00:11 UTC) from {username='slave'} at 127.0.0.1:50624
I20250812 20:00:11.319824 16759 test_util.cc:276] Using random seed: 2094959089
I20250812 20:00:11.321499 23683 tablet_replica.cc:331] stopping tablet replica
I20250812 20:00:11.322414 23683 raft_consensus.cc:2241] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 LEADER]: Raft consensus shutting down.
I20250812 20:00:11.322886 23683 raft_consensus.cc:2270] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 20:00:11.354844 23683 ts_tablet_manager.cc:1905] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250812 20:00:11.359372 23140 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52566:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250812 20:00:11.363247 23140 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 20:00:11.369660 23683 ts_tablet_manager.cc:1918] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250812 20:00:11.370003 23683 log.cc:1199] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Deleting WAL directory at /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/wal/wals/a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:11.370707 23683 ts_tablet_manager.cc:1939] T a55ce92090654fa2a8d6f1b255dec9ee P 4f25919645fc4d6a8ca5540893d2a1d5: Deleting consensus metadata
I20250812 20:00:11.373871 23126 catalog_manager.cc:4928] TS 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813): tablet a55ce92090654fa2a8d6f1b255dec9ee (table pre_rebuild [id=a699787e432e4aacbe86ad8c02a2998b]) successfully deleted
I20250812 20:00:11.390352 23398 tablet_service.cc:1468] Processing CreateTablet for tablet 8f06228d15014784b2ebd399cae78e69 (DEFAULT_TABLE table=post_rebuild [id=d93e132198524fabbed367fe365a535c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 20:00:11.390316 23262 tablet_service.cc:1468] Processing CreateTablet for tablet 8f06228d15014784b2ebd399cae78e69 (DEFAULT_TABLE table=post_rebuild [id=d93e132198524fabbed367fe365a535c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 20:00:11.391485 23398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f06228d15014784b2ebd399cae78e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 20:00:11.391571 23262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f06228d15014784b2ebd399cae78e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 20:00:11.394914 23539 tablet_service.cc:1468] Processing CreateTablet for tablet 8f06228d15014784b2ebd399cae78e69 (DEFAULT_TABLE table=post_rebuild [id=d93e132198524fabbed367fe365a535c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 20:00:11.396319 23539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f06228d15014784b2ebd399cae78e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 20:00:11.413807 23690 tablet_bootstrap.cc:492] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda: Bootstrap starting.
I20250812 20:00:11.420087 23691 tablet_bootstrap.cc:492] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5: Bootstrap starting.
I20250812 20:00:11.420564 23692 tablet_bootstrap.cc:492] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b: Bootstrap starting.
I20250812 20:00:11.427093 23692 tablet_bootstrap.cc:654] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b: Neither blocks nor log segments found. Creating new log.
I20250812 20:00:11.431610 23690 tablet_bootstrap.cc:654] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda: Neither blocks nor log segments found. Creating new log.
I20250812 20:00:11.431969 23691 tablet_bootstrap.cc:654] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5: Neither blocks nor log segments found. Creating new log.
I20250812 20:00:11.441270 23692 tablet_bootstrap.cc:492] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b: No bootstrap required, opened a new log
I20250812 20:00:11.441692 23692 ts_tablet_manager.cc:1397] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20250812 20:00:11.447415 23692 raft_consensus.cc:357] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.450867 23691 tablet_bootstrap.cc:492] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5: No bootstrap required, opened a new log
I20250812 20:00:11.451491 23691 ts_tablet_manager.cc:1397] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.003s
I20250812 20:00:11.454584 23691 raft_consensus.cc:357] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.455276 23691 raft_consensus.cc:383] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 20:00:11.455586 23691 raft_consensus.cc:738] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f25919645fc4d6a8ca5540893d2a1d5, State: Initialized, Role: FOLLOWER
I20250812 20:00:11.448343 23692 raft_consensus.cc:383] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 20:00:11.456375 23692 raft_consensus.cc:738] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbaa1e6b9d394f619027de86958dbc0b, State: Initialized, Role: FOLLOWER
I20250812 20:00:11.457266 23692 consensus_queue.cc:260] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.458506 23690 tablet_bootstrap.cc:492] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda: No bootstrap required, opened a new log
I20250812 20:00:11.459017 23690 ts_tablet_manager.cc:1397] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20250812 20:00:11.459318 23691 consensus_queue.cc:260] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.462051 23690 raft_consensus.cc:357] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.462877 23690 raft_consensus.cc:383] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 20:00:11.463151 23690 raft_consensus.cc:738] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60eda1e8d554494c8fddb4793ccbafda, State: Initialized, Role: FOLLOWER
I20250812 20:00:11.463807 23690 consensus_queue.cc:260] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.471766 23692 ts_tablet_manager.cc:1428] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b: Time spent starting tablet: real 0.030s user 0.009s sys 0.000s
I20250812 20:00:11.472911 23691 ts_tablet_manager.cc:1428] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5: Time spent starting tablet: real 0.021s user 0.010s sys 0.001s
W20250812 20:00:11.487428 23464 tablet.cc:2378] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 20:00:11.488380 23690 ts_tablet_manager.cc:1428] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20250812 20:00:11.656536 23699 raft_consensus.cc:491] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 20:00:11.657135 23699 raft_consensus.cc:513] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.659140 23699 leader_election.cc:290] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
I20250812 20:00:11.660612 23418 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f06228d15014784b2ebd399cae78e69" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" is_pre_election: true
I20250812 20:00:11.661255 23418 raft_consensus.cc:2466] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 0.
I20250812 20:00:11.662799 23216 leader_election.cc:304] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f25919645fc4d6a8ca5540893d2a1d5, 60eda1e8d554494c8fddb4793ccbafda; no voters:
I20250812 20:00:11.663866 23699 raft_consensus.cc:2802] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 20:00:11.664314 23699 raft_consensus.cc:491] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 20:00:11.664732 23699 raft_consensus.cc:3058] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 0 FOLLOWER]: Advancing to term 1
I20250812 20:00:11.672720 23699 raft_consensus.cc:513] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.674558 23699 leader_election.cc:290] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 election: Requested vote from peers cbaa1e6b9d394f619027de86958dbc0b (127.16.93.195:42445), 4f25919645fc4d6a8ca5540893d2a1d5 (127.16.93.194:32813)
I20250812 20:00:11.674621 23559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f06228d15014784b2ebd399cae78e69" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbaa1e6b9d394f619027de86958dbc0b" is_pre_election: true
I20250812 20:00:11.675329 23559 raft_consensus.cc:2466] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 0.
I20250812 20:00:11.675598 23418 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f06228d15014784b2ebd399cae78e69" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f25919645fc4d6a8ca5540893d2a1d5"
I20250812 20:00:11.676141 23418 raft_consensus.cc:3058] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 0 FOLLOWER]: Advancing to term 1
I20250812 20:00:11.676754 23559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f06228d15014784b2ebd399cae78e69" candidate_uuid: "60eda1e8d554494c8fddb4793ccbafda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbaa1e6b9d394f619027de86958dbc0b"
I20250812 20:00:11.677138 23559 raft_consensus.cc:3058] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 0 FOLLOWER]: Advancing to term 1
I20250812 20:00:11.681422 23559 raft_consensus.cc:2466] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 1.
I20250812 20:00:11.682317 23215 leader_election.cc:304] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60eda1e8d554494c8fddb4793ccbafda, cbaa1e6b9d394f619027de86958dbc0b; no voters:
I20250812 20:00:11.682672 23418 raft_consensus.cc:2466] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60eda1e8d554494c8fddb4793ccbafda in term 1.
I20250812 20:00:11.682905 23699 raft_consensus.cc:2802] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 1 FOLLOWER]: Leader election won for term 1
I20250812 20:00:11.684294 23699 raft_consensus.cc:695] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [term 1 LEADER]: Becoming Leader. State: Replica: 60eda1e8d554494c8fddb4793ccbafda, State: Running, Role: LEADER
I20250812 20:00:11.685294 23699 consensus_queue.cc:237] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } }
I20250812 20:00:11.694692 23139 catalog_manager.cc:5582] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda reported cstate change: term changed from 0 to 1, leader changed from <none> to 60eda1e8d554494c8fddb4793ccbafda (127.16.93.193). New cstate: current_term: 1 leader_uuid: "60eda1e8d554494c8fddb4793ccbafda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eda1e8d554494c8fddb4793ccbafda" member_type: VOTER last_known_addr { host: "127.16.93.193" port: 44421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 } health_report { overall_health: UNKNOWN } } }
W20250812 20:00:11.710742 23328 tablet.cc:2378] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 20:00:11.716773 23605 tablet.cc:2378] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 20:00:11.853499 23418 raft_consensus.cc:1273] T 8f06228d15014784b2ebd399cae78e69 P 4f25919645fc4d6a8ca5540893d2a1d5 [term 1 FOLLOWER]: Refusing update from remote peer 60eda1e8d554494c8fddb4793ccbafda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 20:00:11.854072 23559 raft_consensus.cc:1273] T 8f06228d15014784b2ebd399cae78e69 P cbaa1e6b9d394f619027de86958dbc0b [term 1 FOLLOWER]: Refusing update from remote peer 60eda1e8d554494c8fddb4793ccbafda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 20:00:11.855432 23702 consensus_queue.cc:1035] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f25919645fc4d6a8ca5540893d2a1d5" member_type: VOTER last_known_addr { host: "127.16.93.194" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250812 20:00:11.856415 23699 consensus_queue.cc:1035] T 8f06228d15014784b2ebd399cae78e69 P 60eda1e8d554494c8fddb4793ccbafda [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbaa1e6b9d394f619027de86958dbc0b" member_type: VOTER last_known_addr { host: "127.16.93.195" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 20:00:11.889302 23711 mvcc.cc:204] Tried to move back new op lower bound from 7188598013336018944 to 7188598012669530112. Current Snapshot: MvccSnapshot[applied={T|T < 7188598013336018944}]
I20250812 20:00:11.889259 23710 mvcc.cc:204] Tried to move back new op lower bound from 7188598013336018944 to 7188598012669530112. Current Snapshot: MvccSnapshot[applied={T|T < 7188598013336018944}]
I20250812 20:00:11.907951 23712 mvcc.cc:204] Tried to move back new op lower bound from 7188598013336018944 to 7188598012669530112. Current Snapshot: MvccSnapshot[applied={T|T < 7188598013336018944}]
I20250812 20:00:16.873440 23262 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 20:00:16.876776 23398 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 20:00:16.878674 23539 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5f9a62973865426fa180a7767cf92324 | 127.16.93.254:44431 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.93.212:33599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
4f25919645fc4d6a8ca5540893d2a1d5 | 127.16.93.194:32813 | HEALTHY | <none> | 0 | 0
60eda1e8d554494c8fddb4793ccbafda | 127.16.93.193:44421 | HEALTHY | <none> | 1 | 0
cbaa1e6b9d394f619027de86958dbc0b | 127.16.93.195:42445 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.16.93.193 | experimental | 127.16.93.193:44421
local_ip_for_outbound_sockets | 127.16.93.194 | experimental | 127.16.93.194:32813
local_ip_for_outbound_sockets | 127.16.93.195 | experimental | 127.16.93.195:42445
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.93.193:44421
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.93.194:32813
server_dump_info_path | /tmp/dist-test-taskrzrhMv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755028623597350-16759-0/minicluster-data/ts-2/data/info.pb | hidden | 127.16.93.195:42445
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.93.212:33599 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250812 20:00:17.093703 16759 log_verifier.cc:126] Checking tablet 8f06228d15014784b2ebd399cae78e69
I20250812 20:00:17.914835 16759 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 8f06228d15014784b2ebd399cae78e69
I20250812 20:00:17.915868 16759 log_verifier.cc:126] Checking tablet a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:17.916191 16759 log_verifier.cc:177] Verified matching terms for 0 ops in tablet a55ce92090654fa2a8d6f1b255dec9ee
I20250812 20:00:17.941928 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 23177
I20250812 20:00:17.987411 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 23331
I20250812 20:00:18.026572 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 23467
I20250812 20:00:18.061654 16759 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskrzrhMv/build/tsan/bin/kudu with pid 23107
2025-08-12T20:00:18Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (37596 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (37597 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (194455 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST