Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250810 13:57:24.044623 19565 test_util.cc:276] Using random seed: 801212116
W20250810 13:57:25.191952 19565 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.106s user 0.430s sys 0.676s
W20250810 13:57:25.192390 19565 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.107s user 0.430s sys 0.676s
I20250810 13:57:25.194618 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:57:25.194854 19565 ts_itest-base.cc:116] --------------
I20250810 13:57:25.195046 19565 ts_itest-base.cc:117] 4 tablet servers
I20250810 13:57:25.195245 19565 ts_itest-base.cc:118] 3 replicas per TS
I20250810 13:57:25.195422 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:57:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:57:25Z Disabled control of system clock
I20250810 13:57:25.236968 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41767
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:41767 with env {}
W20250810 13:57:25.507025 19579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:25.507548 19579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:25.508049 19579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:25.535838 19579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:57:25.536162 19579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:25.536392 19579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:57:25.536630 19579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:57:25.567916 19579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:41767
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:25.569120 19579 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:25.570588 19579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:25.580991 19585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:25.581472 19586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:26.705375 19587 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250810 13:57:26.706441 19588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:26.706447 19579 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:26.709736 19579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:26.712440 19579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:26.713783 19579 hybrid_clock.cc:648] HybridClock initialized: now 1754834246713734 us; error 42 us; skew 500 ppm
I20250810 13:57:26.714576 19579 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:26.720755 19579 webserver.cc:489] Webserver started at http://127.19.27.126:34317/ using document root <none> and password file <none>
I20250810 13:57:26.721671 19579 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:26.721890 19579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:26.722350 19579 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:26.726823 19579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "83fc89432abc449096a818af02933bbe"
format_stamp: "Formatted at 2025-08-10 13:57:26 on dist-test-slave-k4vt"
I20250810 13:57:26.728041 19579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "83fc89432abc449096a818af02933bbe"
format_stamp: "Formatted at 2025-08-10 13:57:26 on dist-test-slave-k4vt"
I20250810 13:57:26.734740 19579 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250810 13:57:26.739665 19595 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:26.740604 19579 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250810 13:57:26.740912 19579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "83fc89432abc449096a818af02933bbe"
format_stamp: "Formatted at 2025-08-10 13:57:26 on dist-test-slave-k4vt"
I20250810 13:57:26.741211 19579 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:26.785799 19579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:26.787161 19579 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:26.787571 19579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:26.852782 19579 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:41767
I20250810 13:57:26.852802 19646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:41767 every 8 connection(s)
I20250810 13:57:26.855213 19579 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:57:26.859790 19647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:26.863566 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 19579
I20250810 13:57:26.863961 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:57:26.877285 19647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Bootstrap starting.
I20250810 13:57:26.882393 19647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:26.884017 19647 log.cc:826] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:26.888278 19647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: No bootstrap required, opened a new log
I20250810 13:57:26.905386 19647 raft_consensus.cc:357] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:26.906021 19647 raft_consensus.cc:383] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:57:26.906236 19647 raft_consensus.cc:738] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fc89432abc449096a818af02933bbe, State: Initialized, Role: FOLLOWER
I20250810 13:57:26.906985 19647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:26.907629 19647 raft_consensus.cc:397] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:57:26.907981 19647 raft_consensus.cc:491] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:57:26.908399 19647 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:26.912459 19647 raft_consensus.cc:513] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:26.913224 19647 leader_election.cc:304] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83fc89432abc449096a818af02933bbe; no voters:
I20250810 13:57:26.914919 19647 leader_election.cc:290] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:57:26.915572 19652 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:57:26.917694 19652 raft_consensus.cc:695] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 LEADER]: Becoming Leader. State: Replica: 83fc89432abc449096a818af02933bbe, State: Running, Role: LEADER
I20250810 13:57:26.918416 19652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:26.919539 19647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:57:26.928676 19654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83fc89432abc449096a818af02933bbe. Latest consensus state: current_term: 1 leader_uuid: "83fc89432abc449096a818af02933bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } } }
I20250810 13:57:26.929334 19654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: This master's current role is: LEADER
I20250810 13:57:26.930269 19653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83fc89432abc449096a818af02933bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } } }
I20250810 13:57:26.930874 19653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: This master's current role is: LEADER
I20250810 13:57:26.932432 19661 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:57:26.940953 19661 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:57:26.956032 19661 catalog_manager.cc:1349] Generated new cluster ID: c1e57f21fbde4c79829360c39efc7b6c
I20250810 13:57:26.956286 19661 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:57:26.970861 19661 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:57:26.972298 19661 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:57:26.986163 19661 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Generated new TSK 0
I20250810 13:57:26.986910 19661 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:57:26.997013 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--builtin_ntp_servers=127.19.27.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 13:57:27.309582 19671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:27.310050 19671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:27.310561 19671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:27.341053 19671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:27.341845 19671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:57:27.373946 19671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:27.375173 19671 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:27.376745 19671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:27.389216 19677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:27.389925 19678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:28.886086 19680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:28.889629 19679 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1496 milliseconds
I20250810 13:57:28.889711 19671 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:28.890838 19671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:28.893321 19671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:28.894826 19671 hybrid_clock.cc:648] HybridClock initialized: now 1754834248894781 us; error 47 us; skew 500 ppm
I20250810 13:57:28.895614 19671 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:28.901558 19671 webserver.cc:489] Webserver started at http://127.19.27.65:43639/ using document root <none> and password file <none>
I20250810 13:57:28.902393 19671 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:28.902588 19671 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:28.903009 19671 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:28.906966 19671 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f964e7de842844b6beaf7b52a1994228"
format_stamp: "Formatted at 2025-08-10 13:57:28 on dist-test-slave-k4vt"
I20250810 13:57:28.908061 19671 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f964e7de842844b6beaf7b52a1994228"
format_stamp: "Formatted at 2025-08-10 13:57:28 on dist-test-slave-k4vt"
I20250810 13:57:28.914918 19671 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250810 13:57:28.919819 19687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:28.920780 19671 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250810 13:57:28.921059 19671 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f964e7de842844b6beaf7b52a1994228"
format_stamp: "Formatted at 2025-08-10 13:57:28 on dist-test-slave-k4vt"
I20250810 13:57:28.921350 19671 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:28.990432 19671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:28.991698 19671 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:28.992085 19671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:28.994431 19671 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:28.998071 19671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:28.998250 19671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:28.998498 19671 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:28.998638 19671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:29.122586 19671 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:41389
I20250810 13:57:29.122692 19799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:41389 every 8 connection(s)
I20250810 13:57:29.125023 19671 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:57:29.129604 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 19671
I20250810 13:57:29.130079 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:57:29.136039 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--builtin_ntp_servers=127.19.27.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:57:29.144385 19800 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:29.144843 19800 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:29.146019 19800 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:29.148542 19612 ts_manager.cc:194] Registered new tserver with Master: f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389)
I20250810 13:57:29.150179 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:57499
W20250810 13:57:29.418736 19804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:29.419196 19804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:29.419708 19804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:29.449308 19804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:29.450093 19804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:57:29.481784 19804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:29.482964 19804 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:29.484443 19804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:29.496603 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:30.153951 19800 heartbeater.cc:499] Master 127.19.27.126:41767 was elected leader, sending a full tablet report...
W20250810 13:57:29.497238 19811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:30.652526 19813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:30.654908 19812 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1153 milliseconds
I20250810 13:57:30.655000 19804 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:30.656351 19804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:30.660408 19804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:30.661859 19804 hybrid_clock.cc:648] HybridClock initialized: now 1754834250661818 us; error 48 us; skew 500 ppm
I20250810 13:57:30.662685 19804 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:30.668716 19804 webserver.cc:489] Webserver started at http://127.19.27.66:45191/ using document root <none> and password file <none>
I20250810 13:57:30.669592 19804 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:30.669813 19804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:30.670279 19804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:30.676151 19804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "78d47bf1c22747f0820790adc433c146"
format_stamp: "Formatted at 2025-08-10 13:57:30 on dist-test-slave-k4vt"
I20250810 13:57:30.677193 19804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "78d47bf1c22747f0820790adc433c146"
format_stamp: "Formatted at 2025-08-10 13:57:30 on dist-test-slave-k4vt"
I20250810 13:57:30.683945 19804 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250810 13:57:30.689101 19820 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:30.690001 19804 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250810 13:57:30.690285 19804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "78d47bf1c22747f0820790adc433c146"
format_stamp: "Formatted at 2025-08-10 13:57:30 on dist-test-slave-k4vt"
I20250810 13:57:30.690579 19804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:30.738655 19804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:30.739974 19804 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:30.740345 19804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:30.742655 19804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:30.746448 19804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:30.746625 19804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:30.746882 19804 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:30.747051 19804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:30.872992 19804 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:35187
I20250810 13:57:30.873056 19932 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:35187 every 8 connection(s)
I20250810 13:57:30.876081 19804 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:57:30.879123 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 19804
I20250810 13:57:30.879573 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 13:57:30.887828 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--builtin_ntp_servers=127.19.27.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:57:30.905254 19933 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:30.905609 19933 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:30.906471 19933 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:30.908428 19612 ts_manager.cc:194] Registered new tserver with Master: 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187)
I20250810 13:57:30.909534 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:49161
W20250810 13:57:31.170578 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:31.171042 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:31.171517 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:31.201897 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:31.202768 19937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:57:31.237113 19937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:31.238314 19937 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:31.239837 19937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:31.250126 19943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:31.912345 19933 heartbeater.cc:499] Master 127.19.27.126:41767 was elected leader, sending a full tablet report...
W20250810 13:57:31.251979 19944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:32.403520 19946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:32.405721 19945 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1150 milliseconds
I20250810 13:57:32.405853 19937 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:32.406960 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:32.409480 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:32.410884 19937 hybrid_clock.cc:648] HybridClock initialized: now 1754834252410849 us; error 38 us; skew 500 ppm
I20250810 13:57:32.411625 19937 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:32.417775 19937 webserver.cc:489] Webserver started at http://127.19.27.67:38487/ using document root <none> and password file <none>
I20250810 13:57:32.418627 19937 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:32.418805 19937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:32.419214 19937 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:32.423303 19937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
format_stamp: "Formatted at 2025-08-10 13:57:32 on dist-test-slave-k4vt"
I20250810 13:57:32.424357 19937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
format_stamp: "Formatted at 2025-08-10 13:57:32 on dist-test-slave-k4vt"
I20250810 13:57:32.430420 19937 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250810 13:57:32.435456 19953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:32.436343 19937 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250810 13:57:32.436628 19937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
format_stamp: "Formatted at 2025-08-10 13:57:32 on dist-test-slave-k4vt"
I20250810 13:57:32.436928 19937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:32.483620 19937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:32.484900 19937 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:32.485282 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:32.487603 19937 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:32.491348 19937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:32.491557 19937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:32.491851 19937 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:32.492002 19937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:32.617235 19937 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:35437
I20250810 13:57:32.617337 20065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:35437 every 8 connection(s)
I20250810 13:57:32.619560 19937 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:57:32.628619 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 19937
I20250810 13:57:32.628974 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 13:57:32.634352 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.68:0
--local_ip_for_outbound_sockets=127.19.27.68
--webserver_interface=127.19.27.68
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--builtin_ntp_servers=127.19.27.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:57:32.643006 20066 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:32.643481 20066 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:32.644683 20066 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:32.647009 19612 ts_manager.cc:194] Registered new tserver with Master: e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437)
I20250810 13:57:32.648706 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:44579
W20250810 13:57:32.915084 20070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:32.915462 20070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:32.915908 20070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:32.945358 20070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:32.946095 20070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.68
I20250810 13:57:32.977483 20070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.27.68
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.68
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:32.978520 20070 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:32.979868 20070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:32.990180 20076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:33.652458 20066 heartbeater.cc:499] Master 127.19.27.126:41767 was elected leader, sending a full tablet report...
W20250810 13:57:32.990434 20077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:34.283614 20078 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1291 milliseconds
W20250810 13:57:34.284127 20079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:34.286773 20070 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.297s user 0.438s sys 0.853s
W20250810 13:57:34.287144 20070 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.297s user 0.438s sys 0.853s
I20250810 13:57:34.287407 20070 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:34.288659 20070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:34.290637 20070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:34.291949 20070 hybrid_clock.cc:648] HybridClock initialized: now 1754834254291910 us; error 40 us; skew 500 ppm
I20250810 13:57:34.292677 20070 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:34.298750 20070 webserver.cc:489] Webserver started at http://127.19.27.68:42519/ using document root <none> and password file <none>
I20250810 13:57:34.299685 20070 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:34.299961 20070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:34.300415 20070 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:34.304404 20070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca"
format_stamp: "Formatted at 2025-08-10 13:57:34 on dist-test-slave-k4vt"
I20250810 13:57:34.305366 20070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca"
format_stamp: "Formatted at 2025-08-10 13:57:34 on dist-test-slave-k4vt"
I20250810 13:57:34.312183 20070 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250810 13:57:34.317517 20087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:34.318552 20070 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 13:57:34.318825 20070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca"
format_stamp: "Formatted at 2025-08-10 13:57:34 on dist-test-slave-k4vt"
I20250810 13:57:34.319114 20070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:34.390008 20070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:34.391373 20070 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:34.391768 20070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:34.393877 20070 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:34.397255 20070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:34.397428 20070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:34.397641 20070 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:34.397778 20070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:34.517836 20070 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.68:42253
I20250810 13:57:34.517932 20199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.68:42253 every 8 connection(s)
I20250810 13:57:34.520216 20070 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250810 13:57:34.530465 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 20070
I20250810 13:57:34.530972 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250810 13:57:34.539554 20200 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:34.539956 20200 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:34.540874 20200 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:34.542573 19612 ts_manager.cc:194] Registered new tserver with Master: 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68:42253)
I20250810 13:57:34.543620 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.68:59723
I20250810 13:57:34.551180 19565 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250810 13:57:34.586390 19612 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33528:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250810 13:57:34.650713 20001 tablet_service.cc:1468] Processing CreateTablet for tablet 6122e6515cc04c6b90051747e671347f (DEFAULT_TABLE table=TestTable [id=6ed45cd71f1042d49dec9baf65795d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:57:34.652940 19868 tablet_service.cc:1468] Processing CreateTablet for tablet 6122e6515cc04c6b90051747e671347f (DEFAULT_TABLE table=TestTable [id=6ed45cd71f1042d49dec9baf65795d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:57:34.653554 20001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6122e6515cc04c6b90051747e671347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:34.654564 19868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6122e6515cc04c6b90051747e671347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:34.654572 19735 tablet_service.cc:1468] Processing CreateTablet for tablet 6122e6515cc04c6b90051747e671347f (DEFAULT_TABLE table=TestTable [id=6ed45cd71f1042d49dec9baf65795d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:57:34.656124 19735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6122e6515cc04c6b90051747e671347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:34.676570 20219 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Bootstrap starting.
I20250810 13:57:34.678355 20220 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: Bootstrap starting.
I20250810 13:57:34.681149 20221 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: Bootstrap starting.
I20250810 13:57:34.684027 20219 tablet_bootstrap.cc:654] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:34.685752 20220 tablet_bootstrap.cc:654] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:34.686244 20221 tablet_bootstrap.cc:654] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:34.686830 20219 log.cc:826] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:34.687772 20221 log.cc:826] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:34.687947 20220 log.cc:826] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:34.692312 20219 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: No bootstrap required, opened a new log
I20250810 13:57:34.692693 20219 ts_tablet_manager.cc:1397] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250810 13:57:34.693204 20220 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: No bootstrap required, opened a new log
I20250810 13:57:34.693519 20221 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: No bootstrap required, opened a new log
I20250810 13:57:34.693657 20220 ts_tablet_manager.cc:1397] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20250810 13:57:34.693960 20221 ts_tablet_manager.cc:1397] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20250810 13:57:34.708232 20219 raft_consensus.cc:357] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.708793 20219 raft_consensus.cc:383] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:57:34.709031 20219 raft_consensus.cc:738] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f964e7de842844b6beaf7b52a1994228, State: Initialized, Role: FOLLOWER
I20250810 13:57:34.709664 20219 consensus_queue.cc:260] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.713407 20219 ts_tablet_manager.cc:1428] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Time spent starting tablet: real 0.021s user 0.015s sys 0.005s
I20250810 13:57:34.716892 20221 raft_consensus.cc:357] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.716919 20220 raft_consensus.cc:357] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.717645 20220 raft_consensus.cc:383] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:57:34.717674 20221 raft_consensus.cc:383] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:57:34.717947 20220 raft_consensus.cc:738] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d47bf1c22747f0820790adc433c146, State: Initialized, Role: FOLLOWER
I20250810 13:57:34.717954 20221 raft_consensus.cc:738] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b39720577a4cb4af4e172d92a4ac1d, State: Initialized, Role: FOLLOWER
I20250810 13:57:34.718775 20221 consensus_queue.cc:260] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.718772 20220 consensus_queue.cc:260] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.722656 20221 ts_tablet_manager.cc:1428] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: Time spent starting tablet: real 0.028s user 0.024s sys 0.006s
I20250810 13:57:34.726130 20220 ts_tablet_manager.cc:1428] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: Time spent starting tablet: real 0.032s user 0.029s sys 0.003s
I20250810 13:57:34.859650 20227 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:57:34.860143 20227 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.862298 20227 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437), f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389)
W20250810 13:57:34.875260 20067 tablet.cc:2378] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:57:34.877496 20021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "78d47bf1c22747f0820790adc433c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" is_pre_election: true
I20250810 13:57:34.877539 19755 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "78d47bf1c22747f0820790adc433c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f964e7de842844b6beaf7b52a1994228" is_pre_election: true
I20250810 13:57:34.878309 20021 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d47bf1c22747f0820790adc433c146 in term 0.
I20250810 13:57:34.878309 19755 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d47bf1c22747f0820790adc433c146 in term 0.
I20250810 13:57:34.879662 19821 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d47bf1c22747f0820790adc433c146, e2b39720577a4cb4af4e172d92a4ac1d; no voters:
I20250810 13:57:34.880354 20227 raft_consensus.cc:2802] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 13:57:34.880726 20227 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:57:34.881048 20227 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:34.884956 20227 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:34.886220 20227 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [CANDIDATE]: Term 1 election: Requested vote from peers e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437), f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389)
W20250810 13:57:34.886309 19801 tablet.cc:2378] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:57:34.886790 20021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "78d47bf1c22747f0820790adc433c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
I20250810 13:57:34.887187 20021 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:34.888088 19755 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "78d47bf1c22747f0820790adc433c146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f964e7de842844b6beaf7b52a1994228"
I20250810 13:57:34.888499 19755 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:34.892913 20021 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d47bf1c22747f0820790adc433c146 in term 1.
I20250810 13:57:34.893725 19821 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d47bf1c22747f0820790adc433c146, e2b39720577a4cb4af4e172d92a4ac1d; no voters:
I20250810 13:57:34.894232 19755 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d47bf1c22747f0820790adc433c146 in term 1.
I20250810 13:57:34.894371 20227 raft_consensus.cc:2802] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:57:34.896353 20227 raft_consensus.cc:695] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [term 1 LEADER]: Becoming Leader. State: Replica: 78d47bf1c22747f0820790adc433c146, State: Running, Role: LEADER
I20250810 13:57:34.897145 20227 consensus_queue.cc:237] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
W20250810 13:57:34.902186 19934 tablet.cc:2378] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:57:34.909716 19611 catalog_manager.cc:5582] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78d47bf1c22747f0820790adc433c146 (127.19.27.66). New cstate: current_term: 1 leader_uuid: "78d47bf1c22747f0820790adc433c146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } health_report { overall_health: HEALTHY } } }
I20250810 13:57:34.941898 19565 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250810 13:57:34.944968 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f964e7de842844b6beaf7b52a1994228 to finish bootstrapping
I20250810 13:57:34.957278 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 78d47bf1c22747f0820790adc433c146 to finish bootstrapping
I20250810 13:57:34.966524 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e2b39720577a4cb4af4e172d92a4ac1d to finish bootstrapping
I20250810 13:57:34.975317 19565 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250810 13:57:34.977994 19565 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "6122e6515cc04c6b90051747e671347f"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
rpc_addresses {
host: "127.19.27.67"
port: 35437
}
}
ts_infos {
permanent_uuid: "f964e7de842844b6beaf7b52a1994228"
rpc_addresses {
host: "127.19.27.65"
port: 41389
}
}
ts_infos {
permanent_uuid: "78d47bf1c22747f0820790adc433c146"
rpc_addresses {
host: "127.19.27.66"
port: 35187
}
}
I20250810 13:57:35.394850 20227 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:57:35.409883 20227 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P 78d47bf1c22747f0820790adc433c146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:57:35.412580 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 19804
W20250810 13:57:35.438577 19955 connection.cc:537] server connection from 127.19.27.66:56147 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250810 13:57:35.439184 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 19579
W20250810 13:57:35.439105 19596 connection.cc:537] server connection from 127.19.27.66:49161 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250810 13:57:35.463989 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41767
--webserver_interface=127.19.27.126
--webserver_port=34317
--builtin_ntp_servers=127.19.27.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:41767 with env {}
I20250810 13:57:35.546058 20200 heartbeater.cc:499] Master 127.19.27.126:41767 was elected leader, sending a full tablet report...
W20250810 13:57:35.549636 20200 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41767 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41767: connect: Connection refused (error 111)
W20250810 13:57:35.752455 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:35.752986 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:35.753441 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:35.781591 20242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:57:35.781824 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:35.782027 20242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:57:35.782209 20242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:57:35.813414 20242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:41767
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=34317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:35.814482 20242 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:35.815851 20242 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:35.825198 20249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:36.432864 19800 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41767 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41767: connect: Connection refused (error 111)
W20250810 13:57:36.439733 20066 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41767 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41767: connect: Connection refused (error 111)
I20250810 13:57:36.917856 20259 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d47bf1c22747f0820790adc433c146)
I20250810 13:57:36.919903 20259 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:36.921365 20258 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d47bf1c22747f0820790adc433c146)
I20250810 13:57:36.922065 20258 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:36.926643 20259 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389), 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187)
I20250810 13:57:36.950807 20258 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437), 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187)
W20250810 13:57:36.954005 19690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.27.66:35187: connect: Connection refused (error 111)
W20250810 13:57:36.982237 19690 leader_election.cc:336] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187): Network error: Client connection negotiation failed: client connection to 127.19.27.66:35187: connect: Connection refused (error 111)
W20250810 13:57:36.984728 19956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.27.66:35187: connect: Connection refused (error 111)
I20250810 13:57:36.996323 19755 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f964e7de842844b6beaf7b52a1994228" is_pre_election: true
I20250810 13:57:36.996968 19755 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b39720577a4cb4af4e172d92a4ac1d in term 1.
I20250810 13:57:36.998508 19954 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2b39720577a4cb4af4e172d92a4ac1d, f964e7de842844b6beaf7b52a1994228; no voters:
I20250810 13:57:36.999400 20259 raft_consensus.cc:2802] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250810 13:57:36.999742 20259 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78d47bf1c22747f0820790adc433c146)
I20250810 13:57:37.000120 20259 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:57:37.002882 20021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "f964e7de842844b6beaf7b52a1994228" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" is_pre_election: true
W20250810 13:57:37.005236 19956 leader_election.cc:336] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187): Network error: Client connection negotiation failed: client connection to 127.19.27.66:35187: connect: Connection refused (error 111)
I20250810 13:57:37.006533 20259 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:37.007884 20021 raft_consensus.cc:2391] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f964e7de842844b6beaf7b52a1994228 in current term 2: Already voted for candidate e2b39720577a4cb4af4e172d92a4ac1d in this term.
I20250810 13:57:37.012096 19688 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f964e7de842844b6beaf7b52a1994228; no voters: 78d47bf1c22747f0820790adc433c146, e2b39720577a4cb4af4e172d92a4ac1d
I20250810 13:57:37.013477 20258 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:57:37.021955 20258 raft_consensus.cc:2747] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250810 13:57:37.024397 19755 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f964e7de842844b6beaf7b52a1994228"
W20250810 13:57:37.031057 19956 leader_election.cc:336] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187): Network error: Client connection negotiation failed: client connection to 127.19.27.66:35187: connect: Connection refused (error 111)
I20250810 13:57:37.039710 19755 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b39720577a4cb4af4e172d92a4ac1d in term 2.
I20250810 13:57:37.040382 20259 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [CANDIDATE]: Term 2 election: Requested vote from peers f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389), 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187)
I20250810 13:57:37.044188 19954 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e2b39720577a4cb4af4e172d92a4ac1d, f964e7de842844b6beaf7b52a1994228; no voters: 78d47bf1c22747f0820790adc433c146
I20250810 13:57:37.047186 20266 raft_consensus.cc:2802] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 FOLLOWER]: Leader election won for term 2
I20250810 13:57:37.047745 20266 raft_consensus.cc:695] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 LEADER]: Becoming Leader. State: Replica: e2b39720577a4cb4af4e172d92a4ac1d, State: Running, Role: LEADER
I20250810 13:57:37.051280 20266 consensus_queue.cc:237] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
W20250810 13:57:35.825764 20250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:37.229931 20252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:37.233516 20251 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1408 milliseconds
W20250810 13:57:37.234395 20242 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.409s user 0.446s sys 0.942s
W20250810 13:57:37.234638 20242 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.409s user 0.446s sys 0.942s
I20250810 13:57:37.234855 20242 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:37.236002 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:37.238518 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:37.239934 20242 hybrid_clock.cc:648] HybridClock initialized: now 1754834257239911 us; error 38 us; skew 500 ppm
I20250810 13:57:37.240676 20242 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:37.247404 20242 webserver.cc:489] Webserver started at http://127.19.27.126:34317/ using document root <none> and password file <none>
I20250810 13:57:37.248312 20242 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:37.248549 20242 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:37.256513 20242 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20250810 13:57:37.261099 20272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:37.262172 20242 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250810 13:57:37.262431 20242 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "83fc89432abc449096a818af02933bbe"
format_stamp: "Formatted at 2025-08-10 13:57:26 on dist-test-slave-k4vt"
I20250810 13:57:37.264258 20242 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:37.337755 20242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:37.339304 20242 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:37.339754 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:37.409963 20242 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:41767
I20250810 13:57:37.410027 20324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:41767 every 8 connection(s)
I20250810 13:57:37.412696 20242 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:57:37.422052 20325 sys_catalog.cc:263] Verifying existing consensus state
I20250810 13:57:37.422318 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 20242
I20250810 13:57:37.422744 19565 kudu-admin-test.cc:735] Forcing unsafe config change on tserver f964e7de842844b6beaf7b52a1994228
I20250810 13:57:37.426298 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Bootstrap starting.
I20250810 13:57:37.443576 19755 raft_consensus.cc:1273] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 2 FOLLOWER]: Refusing update from remote peer e2b39720577a4cb4af4e172d92a4ac1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250810 13:57:37.445741 20266 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250810 13:57:37.465411 19800 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:37.481881 20325 log.cc:826] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Log is configured to *not* fsync() on all Append() calls
W20250810 13:57:37.489964 19956 consensus_peers.cc:489] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d -> Peer 78d47bf1c22747f0820790adc433c146 (127.19.27.66:35187): Couldn't send request to peer 78d47bf1c22747f0820790adc433c146. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.66:35187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 13:57:37.504014 20066 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:37.515830 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:57:37.516595 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Bootstrap complete.
I20250810 13:57:37.533874 20325 raft_consensus.cc:357] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:37.535856 20325 raft_consensus.cc:738] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fc89432abc449096a818af02933bbe, State: Initialized, Role: FOLLOWER
I20250810 13:57:37.536510 20325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:37.536973 20325 raft_consensus.cc:397] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:57:37.537211 20325 raft_consensus.cc:491] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:57:37.537493 20325 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:57:37.543195 20325 raft_consensus.cc:513] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:37.543732 20325 leader_election.cc:304] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83fc89432abc449096a818af02933bbe; no voters:
I20250810 13:57:37.545841 20325 leader_election.cc:290] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 13:57:37.546188 20334 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 2 FOLLOWER]: Leader election won for term 2
I20250810 13:57:37.548375 20334 raft_consensus.cc:695] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [term 2 LEADER]: Becoming Leader. State: Replica: 83fc89432abc449096a818af02933bbe, State: Running, Role: LEADER
I20250810 13:57:37.549248 20334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } }
I20250810 13:57:37.551362 20325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:57:37.556013 20335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83fc89432abc449096a818af02933bbe. Latest consensus state: current_term: 2 leader_uuid: "83fc89432abc449096a818af02933bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } } }
I20250810 13:57:37.558429 20335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: This master's current role is: LEADER
I20250810 13:57:37.557718 20336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83fc89432abc449096a818af02933bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83fc89432abc449096a818af02933bbe" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41767 } } }
I20250810 13:57:37.558845 20336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe [sys.catalog]: This master's current role is: LEADER
I20250810 13:57:37.577626 20340 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:57:37.585160 20200 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41767
I20250810 13:57:37.585773 20200 heartbeater.cc:499] Master 127.19.27.126:41767 was elected leader, sending a full tablet report...
I20250810 13:57:37.597795 20340 catalog_manager.cc:671] Loaded metadata for table TestTable [id=6ed45cd71f1042d49dec9baf65795d2f]
I20250810 13:57:37.604516 20340 tablet_loader.cc:96] loaded metadata for tablet 6122e6515cc04c6b90051747e671347f (table TestTable [id=6ed45cd71f1042d49dec9baf65795d2f])
I20250810 13:57:37.605880 20340 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:57:37.610174 20340 catalog_manager.cc:1261] Loaded cluster ID: c1e57f21fbde4c79829360c39efc7b6c
I20250810 13:57:37.610461 20340 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:57:37.617383 20340 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:57:37.621953 20340 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 83fc89432abc449096a818af02933bbe: Loaded TSK: 0
I20250810 13:57:37.623379 20340 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 13:57:37.795081 20327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:37.795666 20327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:37.825815 20327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250810 13:57:38.496286 20290 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f964e7de842844b6beaf7b52a1994228" instance_seqno: 1754834249093634) as {username='slave'} at 127.19.27.65:54315; Asking this server to re-register.
I20250810 13:57:38.498231 19800 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:38.498934 19800 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:38.502839 20289 ts_manager.cc:194] Registered new tserver with Master: f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389)
I20250810 13:57:38.511240 20288 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" instance_seqno: 1754834252588994) as {username='slave'} at 127.19.27.67:56389; Asking this server to re-register.
I20250810 13:57:38.512787 20066 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:38.513376 20066 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:38.516629 20288 ts_manager.cc:194] Registered new tserver with Master: e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437)
I20250810 13:57:38.518751 20289 catalog_manager.cc:5582] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 reported cstate change: term changed from 1 to 2, leader changed from 78d47bf1c22747f0820790adc433c146 (127.19.27.66) to e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67). New cstate: current_term: 2 leader_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } } }
I20250810 13:57:38.589443 20200 heartbeater.cc:499] Master 127.19.27.126:41767 was elected leader, sending a full tablet report...
I20250810 13:57:38.591550 20290 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" instance_seqno: 1754834254490151) as {username='slave'} at 127.19.27.68:35429; Asking this server to re-register.
I20250810 13:57:38.592875 20200 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:38.593592 20200 heartbeater.cc:507] Master 127.19.27.126:41767 requested a full tablet report, sending...
I20250810 13:57:38.595705 20290 ts_manager.cc:194] Registered new tserver with Master: 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68:42253)
W20250810 13:57:39.175920 20327 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.311s user 0.405s sys 0.825s
W20250810 13:57:39.176371 20327 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.311s user 0.410s sys 0.825s
I20250810 13:57:39.243279 19754 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "f964e7de842844b6beaf7b52a1994228"
tablet_id: "6122e6515cc04c6b90051747e671347f"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
}
peers {
permanent_uuid: "f964e7de842844b6beaf7b52a1994228"
}
}
from {username='slave'} at 127.0.0.1:56994
W20250810 13:57:39.244426 19754 raft_consensus.cc:2216] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true
I20250810 13:57:39.245256 19754 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 2 FOLLOWER]: Advancing to term 3
I20250810 13:57:39.695122 19754 raft_consensus.cc:1238] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 3 FOLLOWER]: Rejecting Update request from peer e2b39720577a4cb4af4e172d92a4ac1d for earlier term 2. Current term is 3. Ops: []
I20250810 13:57:39.696238 20385 consensus_queue.cc:1046] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250810 13:57:39.697538 20386 raft_consensus.cc:3053] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 LEADER]: Stepping down as leader of term 2
I20250810 13:57:39.697767 20386 raft_consensus.cc:738] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2b39720577a4cb4af4e172d92a4ac1d, State: Running, Role: LEADER
I20250810 13:57:39.698361 20386 consensus_queue.cc:260] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } }
I20250810 13:57:39.699517 20386 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 2 FOLLOWER]: Advancing to term 3
I20250810 13:57:40.751245 20389 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250810 13:57:40.751624 20389 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true
I20250810 13:57:40.752666 20389 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437)
I20250810 13:57:40.753599 20021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "f964e7de842844b6beaf7b52a1994228" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" is_pre_election: true
I20250810 13:57:40.754024 20021 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f964e7de842844b6beaf7b52a1994228 in term 3.
I20250810 13:57:40.754906 19688 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e2b39720577a4cb4af4e172d92a4ac1d, f964e7de842844b6beaf7b52a1994228; no voters:
I20250810 13:57:40.755447 20389 raft_consensus.cc:2802] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250810 13:57:40.755684 20389 raft_consensus.cc:491] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250810 13:57:40.755963 20389 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 3 FOLLOWER]: Advancing to term 4
I20250810 13:57:40.759645 20389 raft_consensus.cc:513] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true
I20250810 13:57:40.760519 20389 leader_election.cc:290] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 4 election: Requested vote from peers e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67:35437)
I20250810 13:57:40.761253 20021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6122e6515cc04c6b90051747e671347f" candidate_uuid: "f964e7de842844b6beaf7b52a1994228" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
I20250810 13:57:40.761577 20021 raft_consensus.cc:3058] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 3 FOLLOWER]: Advancing to term 4
I20250810 13:57:40.765267 20021 raft_consensus.cc:2466] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f964e7de842844b6beaf7b52a1994228 in term 4.
I20250810 13:57:40.765887 19688 leader_election.cc:304] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e2b39720577a4cb4af4e172d92a4ac1d, f964e7de842844b6beaf7b52a1994228; no voters:
I20250810 13:57:40.766338 20389 raft_consensus.cc:2802] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 4 FOLLOWER]: Leader election won for term 4
I20250810 13:57:40.767026 20389 raft_consensus.cc:695] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 4 LEADER]: Becoming Leader. State: Replica: f964e7de842844b6beaf7b52a1994228, State: Running, Role: LEADER
I20250810 13:57:40.767710 20389 consensus_queue.cc:237] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true
I20250810 13:57:40.772779 20288 catalog_manager.cc:5582] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 reported cstate change: term changed from 2 to 4, leader changed from e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67) to f964e7de842844b6beaf7b52a1994228 (127.19.27.65), now has a pending config: VOTER e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67), VOTER f964e7de842844b6beaf7b52a1994228 (127.19.27.65). New cstate: current_term: 4 leader_uuid: "f964e7de842844b6beaf7b52a1994228" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78d47bf1c22747f0820790adc433c146" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35187 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true }
I20250810 13:57:41.275120 20021 raft_consensus.cc:1273] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Refusing update from remote peer f964e7de842844b6beaf7b52a1994228: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250810 13:57:41.276549 20397 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250810 13:57:41.287258 20400 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 78d47bf1c22747f0820790adc433c146 (127.19.27.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true }
I20250810 13:57:41.288561 20021 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 78d47bf1c22747f0820790adc433c146 (127.19.27.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } unsafe_config_change: true }
I20250810 13:57:41.296916 20288 catalog_manager.cc:5582] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 reported cstate change: config changed from index -1 to 3, VOTER 78d47bf1c22747f0820790adc433c146 (127.19.27.66) evicted, no longer has a pending config: VOTER e2b39720577a4cb4af4e172d92a4ac1d (127.19.27.67), VOTER f964e7de842844b6beaf7b52a1994228 (127.19.27.65). New cstate: current_term: 4 leader_uuid: "f964e7de842844b6beaf7b52a1994228" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250810 13:57:41.305327 20288 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 6122e6515cc04c6b90051747e671347f on TS 78d47bf1c22747f0820790adc433c146: Not found: failed to reset TS proxy: Could not find TS for UUID 78d47bf1c22747f0820790adc433c146
I20250810 13:57:41.320277 19754 consensus_queue.cc:237] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: NON_VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: true } } unsafe_config_change: true
I20250810 13:57:41.324645 20021 raft_consensus.cc:1273] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Refusing update from remote peer f964e7de842844b6beaf7b52a1994228: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250810 13:57:41.325917 20400 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250810 13:57:41.330770 20397 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: NON_VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: true } } unsafe_config_change: true }
I20250810 13:57:41.332319 20021 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: NON_VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: true } } unsafe_config_change: true }
W20250810 13:57:41.334827 19691 consensus_peers.cc:489] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 -> Peer 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68:42253): Couldn't send request to peer 0a8efb7f3e724a1288d5a60be1f6b5ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6122e6515cc04c6b90051747e671347f. This is attempt 1: this message will repeat every 5th retry.
I20250810 13:57:41.339103 20274 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6122e6515cc04c6b90051747e671347f with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250810 13:57:41.341611 20289 catalog_manager.cc:5582] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 reported cstate change: config changed from index 3 to 5, NON_VOTER 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) added. New cstate: current_term: 4 leader_uuid: "f964e7de842844b6beaf7b52a1994228" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: NON_VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250810 13:57:41.358305 20275 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 6122e6515cc04c6b90051747e671347f on TS 78d47bf1c22747f0820790adc433c146 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 78d47bf1c22747f0820790adc433c146
I20250810 13:57:41.876699 20410 ts_tablet_manager.cc:927] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Initiating tablet copy from peer f964e7de842844b6beaf7b52a1994228 (127.19.27.65:41389)
I20250810 13:57:41.878547 20410 tablet_copy_client.cc:323] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: tablet copy: Beginning tablet copy session from remote peer at address 127.19.27.65:41389
I20250810 13:57:41.887828 19775 tablet_copy_service.cc:140] P f964e7de842844b6beaf7b52a1994228: Received BeginTabletCopySession request for tablet 6122e6515cc04c6b90051747e671347f from peer 0a8efb7f3e724a1288d5a60be1f6b5ca ({username='slave'} at 127.19.27.68:46995)
I20250810 13:57:41.888247 19775 tablet_copy_service.cc:161] P f964e7de842844b6beaf7b52a1994228: Beginning new tablet copy session on tablet 6122e6515cc04c6b90051747e671347f from peer 0a8efb7f3e724a1288d5a60be1f6b5ca at {username='slave'} at 127.19.27.68:46995: session id = 0a8efb7f3e724a1288d5a60be1f6b5ca-6122e6515cc04c6b90051747e671347f
I20250810 13:57:41.892637 19775 tablet_copy_source_session.cc:215] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 13:57:41.896813 20410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6122e6515cc04c6b90051747e671347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:41.913990 20410 tablet_copy_client.cc:806] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: tablet copy: Starting download of 0 data blocks...
I20250810 13:57:41.914467 20410 tablet_copy_client.cc:670] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: tablet copy: Starting download of 1 WAL segments...
I20250810 13:57:41.917734 20410 tablet_copy_client.cc:538] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 13:57:41.923082 20410 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Bootstrap starting.
I20250810 13:57:41.933660 20410 log.cc:826] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:41.943079 20410 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:57:41.943665 20410 tablet_bootstrap.cc:492] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Bootstrap complete.
I20250810 13:57:41.944166 20410 ts_tablet_manager.cc:1397] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Time spent bootstrapping tablet: real 0.021s user 0.023s sys 0.000s
I20250810 13:57:41.958967 20410 raft_consensus.cc:357] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: NON_VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: true } } unsafe_config_change: true
I20250810 13:57:41.959669 20410 raft_consensus.cc:738] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 0a8efb7f3e724a1288d5a60be1f6b5ca, State: Initialized, Role: LEARNER
I20250810 13:57:41.960181 20410 consensus_queue.cc:260] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: NON_VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: true } } unsafe_config_change: true
I20250810 13:57:41.962834 20410 ts_tablet_manager.cc:1428] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20250810 13:57:41.964264 19775 tablet_copy_service.cc:342] P f964e7de842844b6beaf7b52a1994228: Request end of tablet copy session 0a8efb7f3e724a1288d5a60be1f6b5ca-6122e6515cc04c6b90051747e671347f received from {username='slave'} at 127.19.27.68:46995
I20250810 13:57:41.964617 19775 tablet_copy_service.cc:434] P f964e7de842844b6beaf7b52a1994228: ending tablet copy session 0a8efb7f3e724a1288d5a60be1f6b5ca-6122e6515cc04c6b90051747e671347f on tablet 6122e6515cc04c6b90051747e671347f with peer 0a8efb7f3e724a1288d5a60be1f6b5ca
I20250810 13:57:42.299557 20155 raft_consensus.cc:1215] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250810 13:57:42.524595 20275 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 6122e6515cc04c6b90051747e671347f on TS 78d47bf1c22747f0820790adc433c146 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 78d47bf1c22747f0820790adc433c146
I20250810 13:57:42.822126 20417 raft_consensus.cc:1062] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228: attempting to promote NON_VOTER 0a8efb7f3e724a1288d5a60be1f6b5ca to VOTER
I20250810 13:57:42.823518 20417 consensus_queue.cc:237] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: false } } unsafe_config_change: true
I20250810 13:57:42.828094 20155 raft_consensus.cc:1273] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca [term 4 LEARNER]: Refusing update from remote peer f964e7de842844b6beaf7b52a1994228: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250810 13:57:42.828365 20021 raft_consensus.cc:1273] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Refusing update from remote peer f964e7de842844b6beaf7b52a1994228: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250810 13:57:42.829054 20417 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250810 13:57:42.829790 20420 consensus_queue.cc:1035] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250810 13:57:42.835497 20417 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: false } } unsafe_config_change: true }
I20250810 13:57:42.836762 20021 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P e2b39720577a4cb4af4e172d92a4ac1d [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: false } } unsafe_config_change: true }
I20250810 13:57:42.837862 20155 raft_consensus.cc:2953] T 6122e6515cc04c6b90051747e671347f P 0a8efb7f3e724a1288d5a60be1f6b5ca [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: false } } unsafe_config_change: true }
I20250810 13:57:42.845863 20288 catalog_manager.cc:5582] T 6122e6515cc04c6b90051747e671347f P f964e7de842844b6beaf7b52a1994228 reported cstate change: config changed from index 5 to 6, 0a8efb7f3e724a1288d5a60be1f6b5ca (127.19.27.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "f964e7de842844b6beaf7b52a1994228" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f964e7de842844b6beaf7b52a1994228" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 42253 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250810 13:57:42.950268 19565 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250810 13:57:42.963078 19565 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "6122e6515cc04c6b90051747e671347f"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "e2b39720577a4cb4af4e172d92a4ac1d"
rpc_addresses {
host: "127.19.27.67"
port: 35437
}
}
ts_infos {
permanent_uuid: "f964e7de842844b6beaf7b52a1994228"
rpc_addresses {
host: "127.19.27.65"
port: 41389
}
}
ts_infos {
permanent_uuid: "0a8efb7f3e724a1288d5a60be1f6b5ca"
rpc_addresses {
host: "127.19.27.68"
port: 42253
}
}
I20250810 13:57:42.965397 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 19671
I20250810 13:57:42.990542 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 19937
I20250810 13:57:43.013557 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 20070
I20250810 13:57:43.034950 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 20242
2025-08-10T13:57:43Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19050 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250810 13:57:43.093349 19565 test_util.cc:276] Using random seed: 820260923
I20250810 13:57:43.098671 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:57:43.098809 19565 ts_itest-base.cc:116] --------------
I20250810 13:57:43.098922 19565 ts_itest-base.cc:117] 3 tablet servers
I20250810 13:57:43.099020 19565 ts_itest-base.cc:118] 3 replicas per TS
I20250810 13:57:43.099113 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:57:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:57:43Z Disabled control of system clock
I20250810 13:57:43.133451 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:36749
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:36749
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250810 13:57:43.420032 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:43.420615 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:43.420984 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:43.449970 20439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:57:43.450238 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:43.450456 20439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:57:43.450635 20439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:57:43.484416 20439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:36749
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:36749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:43.485651 20439 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:43.487178 20439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:43.498411 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:43.498412 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:44.759728 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:44.762257 20447 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1260 milliseconds
W20250810 13:57:44.763301 20439 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.265s user 0.432s sys 0.828s
W20250810 13:57:44.763690 20439 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.266s user 0.432s sys 0.828s
I20250810 13:57:44.764012 20439 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:44.765517 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:44.768851 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:44.770326 20439 hybrid_clock.cc:648] HybridClock initialized: now 1754834264770260 us; error 50 us; skew 500 ppm
I20250810 13:57:44.771484 20439 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:44.780115 20439 webserver.cc:489] Webserver started at http://127.19.27.126:35749/ using document root <none> and password file <none>
I20250810 13:57:44.781508 20439 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:44.781802 20439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:44.782387 20439 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:44.789036 20439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ae59f9e2d3f949e298decfea57117512"
format_stamp: "Formatted at 2025-08-10 13:57:44 on dist-test-slave-k4vt"
I20250810 13:57:44.790520 20439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ae59f9e2d3f949e298decfea57117512"
format_stamp: "Formatted at 2025-08-10 13:57:44 on dist-test-slave-k4vt"
I20250810 13:57:44.800297 20439 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.006s
I20250810 13:57:44.807843 20456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:44.808951 20439 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250810 13:57:44.809340 20439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ae59f9e2d3f949e298decfea57117512"
format_stamp: "Formatted at 2025-08-10 13:57:44 on dist-test-slave-k4vt"
I20250810 13:57:44.809772 20439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:44.882329 20439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:44.883517 20439 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:44.883913 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:44.946555 20439 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:36749
I20250810 13:57:44.946620 20507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:36749 every 8 connection(s)
I20250810 13:57:44.949079 20439 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:57:44.953496 20508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:44.954586 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 20439
I20250810 13:57:44.955192 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:57:44.976220 20508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512: Bootstrap starting.
I20250810 13:57:44.981700 20508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:44.983134 20508 log.cc:826] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:44.986847 20508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512: No bootstrap required, opened a new log
I20250810 13:57:45.001943 20508 raft_consensus.cc:357] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae59f9e2d3f949e298decfea57117512" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 36749 } }
I20250810 13:57:45.002417 20508 raft_consensus.cc:383] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:57:45.002610 20508 raft_consensus.cc:738] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae59f9e2d3f949e298decfea57117512, State: Initialized, Role: FOLLOWER
I20250810 13:57:45.003230 20508 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae59f9e2d3f949e298decfea57117512" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 36749 } }
I20250810 13:57:45.003684 20508 raft_consensus.cc:397] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:57:45.003970 20508 raft_consensus.cc:491] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:57:45.004233 20508 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:45.008034 20508 raft_consensus.cc:513] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae59f9e2d3f949e298decfea57117512" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 36749 } }
I20250810 13:57:45.008553 20508 leader_election.cc:304] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae59f9e2d3f949e298decfea57117512; no voters:
I20250810 13:57:45.009974 20508 leader_election.cc:290] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:57:45.010617 20513 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:57:45.012652 20513 raft_consensus.cc:695] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [term 1 LEADER]: Becoming Leader. State: Replica: ae59f9e2d3f949e298decfea57117512, State: Running, Role: LEADER
I20250810 13:57:45.013306 20513 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae59f9e2d3f949e298decfea57117512" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 36749 } }
I20250810 13:57:45.014103 20508 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:57:45.019114 20514 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae59f9e2d3f949e298decfea57117512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae59f9e2d3f949e298decfea57117512" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 36749 } } }
I20250810 13:57:45.019475 20515 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae59f9e2d3f949e298decfea57117512. Latest consensus state: current_term: 1 leader_uuid: "ae59f9e2d3f949e298decfea57117512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae59f9e2d3f949e298decfea57117512" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 36749 } } }
I20250810 13:57:45.019840 20514 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [sys.catalog]: This master's current role is: LEADER
I20250810 13:57:45.020044 20515 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512 [sys.catalog]: This master's current role is: LEADER
I20250810 13:57:45.022890 20520 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:57:45.033141 20520 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:57:45.046903 20520 catalog_manager.cc:1349] Generated new cluster ID: ce6d3cca8e944c088efebff19ae1809d
I20250810 13:57:45.047155 20520 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:57:45.058389 20520 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:57:45.059543 20520 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:57:45.072257 20520 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ae59f9e2d3f949e298decfea57117512: Generated new TSK 0
I20250810 13:57:45.072902 20520 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:57:45.081382 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:36749
--builtin_ntp_servers=127.19.27.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250810 13:57:45.358891 20532 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250810 13:57:45.359412 20532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:45.359660 20532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:45.360132 20532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:45.387157 20532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:45.387935 20532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:57:45.419191 20532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:36749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:45.420367 20532 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:45.421846 20532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:45.433183 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:46.834525 20537 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 20532
W20250810 13:57:47.288120 20537 kernel_stack_watchdog.cc:198] Thread 20532 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:57:45.433718 20539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:47.289671 20532 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.855s user 0.002s sys 0.000s
W20250810 13:57:47.290040 20532 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.856s user 0.002s sys 0.000s
W20250810 13:57:47.296957 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:47.299156 20540 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1862 milliseconds
I20250810 13:57:47.299229 20532 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:47.300705 20532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:47.303074 20532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:47.304554 20532 hybrid_clock.cc:648] HybridClock initialized: now 1754834267304486 us; error 59 us; skew 500 ppm
I20250810 13:57:47.305572 20532 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:47.312235 20532 webserver.cc:489] Webserver started at http://127.19.27.65:45471/ using document root <none> and password file <none>
I20250810 13:57:47.313337 20532 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:47.313586 20532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:47.314090 20532 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:47.320108 20532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "41f45fc37ce74af8a4356487fe21aae6"
format_stamp: "Formatted at 2025-08-10 13:57:47 on dist-test-slave-k4vt"
I20250810 13:57:47.321547 20532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "41f45fc37ce74af8a4356487fe21aae6"
format_stamp: "Formatted at 2025-08-10 13:57:47 on dist-test-slave-k4vt"
I20250810 13:57:47.330281 20532 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250810 13:57:47.337275 20548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:47.338340 20532 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 13:57:47.338732 20532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "41f45fc37ce74af8a4356487fe21aae6"
format_stamp: "Formatted at 2025-08-10 13:57:47 on dist-test-slave-k4vt"
I20250810 13:57:47.339141 20532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:47.424044 20532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:47.425401 20532 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:47.425815 20532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:47.428457 20532 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:47.432349 20532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:47.432556 20532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:47.432829 20532 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:47.432984 20532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:47.591038 20532 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:42753
I20250810 13:57:47.591143 20660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:42753 every 8 connection(s)
I20250810 13:57:47.593623 20532 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:57:47.601154 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 20532
I20250810 13:57:47.601573 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:57:47.609493 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:36749
--builtin_ntp_servers=127.19.27.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250810 13:57:47.619906 20661 heartbeater.cc:344] Connected to a master server at 127.19.27.126:36749
I20250810 13:57:47.620401 20661 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:47.621644 20661 heartbeater.cc:507] Master 127.19.27.126:36749 requested a full tablet report, sending...
I20250810 13:57:47.624642 20473 ts_manager.cc:194] Registered new tserver with Master: 41f45fc37ce74af8a4356487fe21aae6 (127.19.27.65:42753)
I20250810 13:57:47.627390 20473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:57731
W20250810 13:57:47.899066 20665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250810 13:57:47.899755 20665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:47.900044 20665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:47.900512 20665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:47.931135 20665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:47.932008 20665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:57:47.980774 20665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:36749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:47.982043 20665 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:47.983415 20665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:47.993980 20671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:48.631523 20661 heartbeater.cc:499] Master 127.19.27.126:36749 was elected leader, sending a full tablet report...
W20250810 13:57:47.995656 20672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:49.157914 20674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:49.159634 20673 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1160 milliseconds
I20250810 13:57:49.159718 20665 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:49.160867 20665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:49.164148 20665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:49.165472 20665 hybrid_clock.cc:648] HybridClock initialized: now 1754834269165432 us; error 49 us; skew 500 ppm
I20250810 13:57:49.166237 20665 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:49.172655 20665 webserver.cc:489] Webserver started at http://127.19.27.66:36163/ using document root <none> and password file <none>
I20250810 13:57:49.173563 20665 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:49.173772 20665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:49.174232 20665 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:49.178471 20665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ae26d3b51e00416281824eba9e481255"
format_stamp: "Formatted at 2025-08-10 13:57:49 on dist-test-slave-k4vt"
I20250810 13:57:49.179546 20665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ae26d3b51e00416281824eba9e481255"
format_stamp: "Formatted at 2025-08-10 13:57:49 on dist-test-slave-k4vt"
I20250810 13:57:49.186566 20665 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250810 13:57:49.192006 20681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:49.192991 20665 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250810 13:57:49.193310 20665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ae26d3b51e00416281824eba9e481255"
format_stamp: "Formatted at 2025-08-10 13:57:49 on dist-test-slave-k4vt"
I20250810 13:57:49.193686 20665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:49.257650 20665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:49.259096 20665 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:49.259519 20665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:49.262001 20665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:49.265762 20665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:49.265959 20665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:49.266219 20665 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:49.266366 20665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:49.395161 20665 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:34277
I20250810 13:57:49.395262 20793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:34277 every 8 connection(s)
I20250810 13:57:49.397619 20665 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:57:49.408167 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 20665
I20250810 13:57:49.408543 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 13:57:49.414466 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:36749
--builtin_ntp_servers=127.19.27.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250810 13:57:49.417122 20794 heartbeater.cc:344] Connected to a master server at 127.19.27.126:36749
I20250810 13:57:49.417629 20794 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:49.419024 20794 heartbeater.cc:507] Master 127.19.27.126:36749 requested a full tablet report, sending...
I20250810 13:57:49.421269 20473 ts_manager.cc:194] Registered new tserver with Master: ae26d3b51e00416281824eba9e481255 (127.19.27.66:34277)
I20250810 13:57:49.422761 20473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:32869
W20250810 13:57:49.707106 20798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250810 13:57:49.707691 20798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:49.707938 20798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:49.708408 20798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:49.737505 20798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:49.738278 20798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:57:49.770845 20798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:36749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:49.772043 20798 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:49.773586 20798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:49.784418 20804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:50.425805 20794 heartbeater.cc:499] Master 127.19.27.126:36749 was elected leader, sending a full tablet report...
W20250810 13:57:51.189636 20803 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 20798
W20250810 13:57:49.785991 20805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:51.436107 20806 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1650 milliseconds
W20250810 13:57:51.436957 20803 kernel_stack_watchdog.cc:198] Thread 20798 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:57:51.437274 20798 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.652s user 0.002s sys 0.000s
W20250810 13:57:51.437703 20798 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.652s user 0.002s sys 0.000s
W20250810 13:57:51.438776 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:51.438846 20798 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:57:51.441828 20798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:51.444396 20798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:51.445850 20798 hybrid_clock.cc:648] HybridClock initialized: now 1754834271445813 us; error 39 us; skew 500 ppm
I20250810 13:57:51.446583 20798 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:51.452562 20798 webserver.cc:489] Webserver started at http://127.19.27.67:41985/ using document root <none> and password file <none>
I20250810 13:57:51.453446 20798 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:51.453632 20798 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:51.454054 20798 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:51.458562 20798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "76440dd996ce40f1a3bc00ca55bbe6e4"
format_stamp: "Formatted at 2025-08-10 13:57:51 on dist-test-slave-k4vt"
I20250810 13:57:51.459656 20798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "76440dd996ce40f1a3bc00ca55bbe6e4"
format_stamp: "Formatted at 2025-08-10 13:57:51 on dist-test-slave-k4vt"
I20250810 13:57:51.466549 20798 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250810 13:57:51.471964 20814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:51.472934 20798 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250810 13:57:51.473235 20798 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "76440dd996ce40f1a3bc00ca55bbe6e4"
format_stamp: "Formatted at 2025-08-10 13:57:51 on dist-test-slave-k4vt"
I20250810 13:57:51.473574 20798 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:51.520409 20798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:51.521808 20798 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:51.522222 20798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:57:51.524670 20798 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:57:51.528648 20798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:57:51.528836 20798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:51.529124 20798 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:57:51.529273 20798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:51.664214 20798 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:33907
I20250810 13:57:51.664376 20926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:33907 every 8 connection(s)
I20250810 13:57:51.666643 20798 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:57:51.674942 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 20798
I20250810 13:57:51.675307 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 13:57:51.689354 20927 heartbeater.cc:344] Connected to a master server at 127.19.27.126:36749
I20250810 13:57:51.689764 20927 heartbeater.cc:461] Registering TS with master...
I20250810 13:57:51.690737 20927 heartbeater.cc:507] Master 127.19.27.126:36749 requested a full tablet report, sending...
I20250810 13:57:51.692955 20472 ts_manager.cc:194] Registered new tserver with Master: 76440dd996ce40f1a3bc00ca55bbe6e4 (127.19.27.67:33907)
I20250810 13:57:51.694340 20472 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:42537
I20250810 13:57:51.694869 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:57:51.730909 20472 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49672:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250810 13:57:51.750217 20472 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 13:57:51.802842 20729 tablet_service.cc:1468] Processing CreateTablet for tablet 2828743d76034558af9b2763ea9aaddf (DEFAULT_TABLE table=TestTable [id=ccadaf5aa9934d088284344e3d12b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:57:51.805197 20729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2828743d76034558af9b2763ea9aaddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:51.809948 20596 tablet_service.cc:1468] Processing CreateTablet for tablet 2828743d76034558af9b2763ea9aaddf (DEFAULT_TABLE table=TestTable [id=ccadaf5aa9934d088284344e3d12b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:57:51.811834 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2828743d76034558af9b2763ea9aaddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:51.813067 20862 tablet_service.cc:1468] Processing CreateTablet for tablet 2828743d76034558af9b2763ea9aaddf (DEFAULT_TABLE table=TestTable [id=ccadaf5aa9934d088284344e3d12b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:57:51.814939 20862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2828743d76034558af9b2763ea9aaddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:57:51.828364 20946 tablet_bootstrap.cc:492] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: Bootstrap starting.
I20250810 13:57:51.833416 20946 tablet_bootstrap.cc:654] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:51.835078 20946 log.cc:826] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:51.838835 20948 tablet_bootstrap.cc:492] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: Bootstrap starting.
I20250810 13:57:51.842276 20949 tablet_bootstrap.cc:492] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: Bootstrap starting.
I20250810 13:57:51.842577 20946 tablet_bootstrap.cc:492] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: No bootstrap required, opened a new log
I20250810 13:57:51.842979 20946 ts_tablet_manager.cc:1397] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20250810 13:57:51.848063 20948 tablet_bootstrap.cc:654] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:51.849977 20949 tablet_bootstrap.cc:654] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: Neither blocks nor log segments found. Creating new log.
I20250810 13:57:51.850456 20948 log.cc:826] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:51.852409 20949 log.cc:826] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: Log is configured to *not* fsync() on all Append() calls
I20250810 13:57:51.858320 20948 tablet_bootstrap.cc:492] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: No bootstrap required, opened a new log
I20250810 13:57:51.858705 20948 ts_tablet_manager.cc:1397] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.013s
I20250810 13:57:51.859234 20949 tablet_bootstrap.cc:492] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: No bootstrap required, opened a new log
I20250810 13:57:51.859692 20949 ts_tablet_manager.cc:1397] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250810 13:57:51.863689 20946 raft_consensus.cc:357] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.864643 20946 raft_consensus.cc:738] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae26d3b51e00416281824eba9e481255, State: Initialized, Role: FOLLOWER
I20250810 13:57:51.865327 20946 consensus_queue.cc:260] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.869438 20946 ts_tablet_manager.cc:1428] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: Time spent starting tablet: real 0.026s user 0.025s sys 0.000s
I20250810 13:57:51.878705 20948 raft_consensus.cc:357] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.879554 20948 raft_consensus.cc:738] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f45fc37ce74af8a4356487fe21aae6, State: Initialized, Role: FOLLOWER
I20250810 13:57:51.880245 20948 consensus_queue.cc:260] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.881798 20949 raft_consensus.cc:357] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.882690 20949 raft_consensus.cc:738] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76440dd996ce40f1a3bc00ca55bbe6e4, State: Initialized, Role: FOLLOWER
I20250810 13:57:51.883415 20949 consensus_queue.cc:260] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.886101 20948 ts_tablet_manager.cc:1428] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: Time spent starting tablet: real 0.027s user 0.022s sys 0.005s
I20250810 13:57:51.888983 20927 heartbeater.cc:499] Master 127.19.27.126:36749 was elected leader, sending a full tablet report...
I20250810 13:57:51.890417 20949 ts_tablet_manager.cc:1428] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: Time spent starting tablet: real 0.030s user 0.021s sys 0.008s
I20250810 13:57:51.900923 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:57:51.904018 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 41f45fc37ce74af8a4356487fe21aae6 to finish bootstrapping
W20250810 13:57:51.906307 20795 tablet.cc:2378] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:57:51.917440 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ae26d3b51e00416281824eba9e481255 to finish bootstrapping
W20250810 13:57:51.921115 20928 tablet.cc:2378] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:57:51.928143 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 76440dd996ce40f1a3bc00ca55bbe6e4 to finish bootstrapping
I20250810 13:57:51.971432 20616 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf"
dest_uuid: "41f45fc37ce74af8a4356487fe21aae6"
from {username='slave'} at 127.0.0.1:32780
I20250810 13:57:51.972043 20616 raft_consensus.cc:491] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250810 13:57:51.972370 20616 raft_consensus.cc:3058] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:51.976588 20616 raft_consensus.cc:513] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:51.978834 20616 leader_election.cc:290] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [CANDIDATE]: Term 1 election: Requested vote from peers ae26d3b51e00416281824eba9e481255 (127.19.27.66:34277), 76440dd996ce40f1a3bc00ca55bbe6e4 (127.19.27.67:33907)
I20250810 13:57:51.987105 19565 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250810 13:57:51.992684 20749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf" candidate_uuid: "41f45fc37ce74af8a4356487fe21aae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ae26d3b51e00416281824eba9e481255"
I20250810 13:57:51.993291 20749 raft_consensus.cc:3058] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:51.997139 20882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf" candidate_uuid: "41f45fc37ce74af8a4356487fe21aae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4"
I20250810 13:57:51.997818 20882 raft_consensus.cc:3058] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:57:51.999460 20749 raft_consensus.cc:2466] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f45fc37ce74af8a4356487fe21aae6 in term 1.
I20250810 13:57:52.000661 20549 leader_election.cc:304] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f45fc37ce74af8a4356487fe21aae6, ae26d3b51e00416281824eba9e481255; no voters:
I20250810 13:57:52.001541 20953 raft_consensus.cc:2802] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:57:52.003257 20882 raft_consensus.cc:2466] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f45fc37ce74af8a4356487fe21aae6 in term 1.
I20250810 13:57:52.003589 20953 raft_consensus.cc:695] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 LEADER]: Becoming Leader. State: Replica: 41f45fc37ce74af8a4356487fe21aae6, State: Running, Role: LEADER
I20250810 13:57:52.004524 20953 consensus_queue.cc:237] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:52.014770 20471 catalog_manager.cc:5582] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41f45fc37ce74af8a4356487fe21aae6 (127.19.27.65). New cstate: current_term: 1 leader_uuid: "41f45fc37ce74af8a4356487fe21aae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } health_report { overall_health: HEALTHY } } }
I20250810 13:57:52.092520 19565 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250810 13:57:52.102813 20662 tablet.cc:2378] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:57:52.298321 19565 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250810 13:57:52.455744 20953 consensus_queue.cc:1035] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:57:52.471181 20953 consensus_queue.cc:1035] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:57:54.284300 20616 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf"
dest_uuid: "41f45fc37ce74af8a4356487fe21aae6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32794
I20250810 13:57:54.284829 20616 raft_consensus.cc:604] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 LEADER]: Received request to transfer leadership
I20250810 13:57:54.389149 20973 raft_consensus.cc:991] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6: : Instructing follower 76440dd996ce40f1a3bc00ca55bbe6e4 to start an election
I20250810 13:57:54.389518 20973 raft_consensus.cc:1079] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 LEADER]: Signalling peer 76440dd996ce40f1a3bc00ca55bbe6e4 to start an election
I20250810 13:57:54.390884 20882 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf"
dest_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4"
from {username='slave'} at 127.19.27.65:41113
I20250810 13:57:54.391407 20882 raft_consensus.cc:491] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250810 13:57:54.391669 20882 raft_consensus.cc:3058] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:57:54.395972 20882 raft_consensus.cc:513] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:54.398226 20882 leader_election.cc:290] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [CANDIDATE]: Term 2 election: Requested vote from peers ae26d3b51e00416281824eba9e481255 (127.19.27.66:34277), 41f45fc37ce74af8a4356487fe21aae6 (127.19.27.65:42753)
I20250810 13:57:54.410856 20616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf" candidate_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "41f45fc37ce74af8a4356487fe21aae6"
I20250810 13:57:54.411492 20616 raft_consensus.cc:3053] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 LEADER]: Stepping down as leader of term 1
I20250810 13:57:54.411703 20616 raft_consensus.cc:738] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 41f45fc37ce74af8a4356487fe21aae6, State: Running, Role: LEADER
I20250810 13:57:54.411676 20749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf" candidate_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae26d3b51e00416281824eba9e481255"
I20250810 13:57:54.412212 20616 consensus_queue.cc:260] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:54.412521 20749 raft_consensus.cc:3058] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:57:54.413075 20616 raft_consensus.cc:3058] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:57:54.417738 20616 raft_consensus.cc:2466] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76440dd996ce40f1a3bc00ca55bbe6e4 in term 2.
I20250810 13:57:54.419095 20817 leader_election.cc:304] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f45fc37ce74af8a4356487fe21aae6, 76440dd996ce40f1a3bc00ca55bbe6e4; no voters:
I20250810 13:57:54.419214 20749 raft_consensus.cc:2466] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76440dd996ce40f1a3bc00ca55bbe6e4 in term 2.
I20250810 13:57:54.421413 20994 raft_consensus.cc:2802] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 13:57:54.422825 20994 raft_consensus.cc:695] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [term 2 LEADER]: Becoming Leader. State: Replica: 76440dd996ce40f1a3bc00ca55bbe6e4, State: Running, Role: LEADER
I20250810 13:57:54.423686 20994 consensus_queue.cc:237] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } }
I20250810 13:57:54.431324 20472 catalog_manager.cc:5582] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 reported cstate change: term changed from 1 to 2, leader changed from 41f45fc37ce74af8a4356487fe21aae6 (127.19.27.65) to 76440dd996ce40f1a3bc00ca55bbe6e4 (127.19.27.67). New cstate: current_term: 2 leader_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76440dd996ce40f1a3bc00ca55bbe6e4" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 33907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 } health_report { overall_health: UNKNOWN } } }
I20250810 13:57:54.867183 20616 raft_consensus.cc:1273] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 2 FOLLOWER]: Refusing update from remote peer 76440dd996ce40f1a3bc00ca55bbe6e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250810 13:57:54.868480 20994 consensus_queue.cc:1035] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f45fc37ce74af8a4356487fe21aae6" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250810 13:57:54.879565 20749 raft_consensus.cc:1273] T 2828743d76034558af9b2763ea9aaddf P ae26d3b51e00416281824eba9e481255 [term 2 FOLLOWER]: Refusing update from remote peer 76440dd996ce40f1a3bc00ca55bbe6e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250810 13:57:54.881386 21003 consensus_queue.cc:1035] T 2828743d76034558af9b2763ea9aaddf P 76440dd996ce40f1a3bc00ca55bbe6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae26d3b51e00416281824eba9e481255" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250810 13:57:57.020695 20616 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2828743d76034558af9b2763ea9aaddf"
dest_uuid: "41f45fc37ce74af8a4356487fe21aae6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32796
I20250810 13:57:57.021366 20616 raft_consensus.cc:604] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 2 FOLLOWER]: Received request to transfer leadership
I20250810 13:57:57.021675 20616 raft_consensus.cc:612] T 2828743d76034558af9b2763ea9aaddf P 41f45fc37ce74af8a4356487fe21aae6 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250810 13:57:58.056740 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 20532
I20250810 13:57:58.082180 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 20665
I20250810 13:57:58.106338 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 20798
I20250810 13:57:58.132141 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 20439
2025-08-10T13:57:58Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (15098 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250810 13:57:58.192718 19565 test_util.cc:276] Using random seed: 835360303
I20250810 13:57:58.196976 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:57:58.197156 19565 ts_itest-base.cc:116] --------------
I20250810 13:57:58.197348 19565 ts_itest-base.cc:117] 3 tablet servers
I20250810 13:57:58.197489 19565 ts_itest-base.cc:118] 3 replicas per TS
I20250810 13:57:58.197633 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:57:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:57:58Z Disabled control of system clock
I20250810 13:57:58.235929 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:43769
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:43769 with env {}
W20250810 13:57:58.565219 21037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:57:58.565877 21037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:57:58.566359 21037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:57:58.598258 21037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:57:58.598623 21037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:57:58.598882 21037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:57:58.599125 21037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:57:58.635748 21037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:43769
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:57:58.637162 21037 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:57:58.638783 21037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:57:58.650085 21043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:58.650300 21044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:57:59.831328 21037 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.181s user 0.388s sys 0.788s
W20250810 13:57:59.831880 21037 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.182s user 0.389s sys 0.788s
W20250810 13:57:59.831992 21046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:57:59.832499 21037 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250810 13:57:59.832501 21045 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1181 milliseconds
I20250810 13:57:59.834048 21037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:57:59.836668 21037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:57:59.837996 21037 hybrid_clock.cc:648] HybridClock initialized: now 1754834279837971 us; error 41 us; skew 500 ppm
I20250810 13:57:59.838790 21037 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:57:59.846057 21037 webserver.cc:489] Webserver started at http://127.19.27.126:45667/ using document root <none> and password file <none>
I20250810 13:57:59.847090 21037 fs_manager.cc:362] Metadata directory not provided
I20250810 13:57:59.847357 21037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:57:59.847862 21037 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:57:59.852367 21037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b625760bf4e846f9b04ad49e5096bb61"
format_stamp: "Formatted at 2025-08-10 13:57:59 on dist-test-slave-k4vt"
I20250810 13:57:59.853574 21037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b625760bf4e846f9b04ad49e5096bb61"
format_stamp: "Formatted at 2025-08-10 13:57:59 on dist-test-slave-k4vt"
I20250810 13:57:59.862215 21037 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20250810 13:57:59.868526 21053 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:57:59.869755 21037 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 13:57:59.870078 21037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b625760bf4e846f9b04ad49e5096bb61"
format_stamp: "Formatted at 2025-08-10 13:57:59 on dist-test-slave-k4vt"
I20250810 13:57:59.870436 21037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:57:59.956701 21037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:57:59.958211 21037 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:57:59.958627 21037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:00.029264 21037 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:43769
I20250810 13:58:00.029356 21104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:43769 every 8 connection(s)
I20250810 13:58:00.032128 21037 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:58:00.037693 21105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:00.041570 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21037
I20250810 13:58:00.042053 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:58:00.062119 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61: Bootstrap starting.
I20250810 13:58:00.068420 21105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:00.070178 21105 log.cc:826] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:00.074959 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61: No bootstrap required, opened a new log
I20250810 13:58:00.092888 21105 raft_consensus.cc:357] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b625760bf4e846f9b04ad49e5096bb61" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 43769 } }
I20250810 13:58:00.093622 21105 raft_consensus.cc:383] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:00.093860 21105 raft_consensus.cc:738] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b625760bf4e846f9b04ad49e5096bb61, State: Initialized, Role: FOLLOWER
I20250810 13:58:00.094718 21105 consensus_queue.cc:260] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b625760bf4e846f9b04ad49e5096bb61" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 43769 } }
I20250810 13:58:00.095285 21105 raft_consensus.cc:397] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:00.095548 21105 raft_consensus.cc:491] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:00.095877 21105 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:00.100421 21105 raft_consensus.cc:513] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b625760bf4e846f9b04ad49e5096bb61" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 43769 } }
I20250810 13:58:00.101141 21105 leader_election.cc:304] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b625760bf4e846f9b04ad49e5096bb61; no voters:
I20250810 13:58:00.102887 21105 leader_election.cc:290] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:58:00.103668 21110 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:00.107445 21110 raft_consensus.cc:695] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [term 1 LEADER]: Becoming Leader. State: Replica: b625760bf4e846f9b04ad49e5096bb61, State: Running, Role: LEADER
I20250810 13:58:00.108264 21105 sys_catalog.cc:564] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:58:00.108652 21110 consensus_queue.cc:237] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b625760bf4e846f9b04ad49e5096bb61" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 43769 } }
I20250810 13:58:00.117769 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b625760bf4e846f9b04ad49e5096bb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b625760bf4e846f9b04ad49e5096bb61" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 43769 } } }
I20250810 13:58:00.118578 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:00.120611 21112 sys_catalog.cc:455] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b625760bf4e846f9b04ad49e5096bb61. Latest consensus state: current_term: 1 leader_uuid: "b625760bf4e846f9b04ad49e5096bb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b625760bf4e846f9b04ad49e5096bb61" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 43769 } } }
I20250810 13:58:00.121410 21112 sys_catalog.cc:458] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:00.123939 21119 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:58:00.136601 21119 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:58:00.151264 21119 catalog_manager.cc:1349] Generated new cluster ID: c0cd416de7ee469a90b599dfa0288be1
I20250810 13:58:00.151623 21119 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:58:00.197891 21119 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:58:00.199465 21119 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:58:00.218925 21119 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b625760bf4e846f9b04ad49e5096bb61: Generated new TSK 0
I20250810 13:58:00.219961 21119 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:58:00.232544 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:43769
--builtin_ntp_servers=127.19.27.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 13:58:00.553920 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:00.554494 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:00.555030 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:00.587520 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:00.588452 21129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:58:00.625802 21129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:00.627166 21129 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:00.628932 21129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:00.642158 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:02.044844 21134 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 21129
W20250810 13:58:00.643568 21136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:02.406591 21129 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.762s user 0.468s sys 1.122s
W20250810 13:58:02.406996 21129 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.764s user 0.468s sys 1.122s
W20250810 13:58:02.407851 21138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:02.410264 21137 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1763 milliseconds
I20250810 13:58:02.410339 21129 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:02.411666 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:02.413894 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:02.415247 21129 hybrid_clock.cc:648] HybridClock initialized: now 1754834282415196 us; error 61 us; skew 500 ppm
I20250810 13:58:02.416098 21129 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:02.422957 21129 webserver.cc:489] Webserver started at http://127.19.27.65:36859/ using document root <none> and password file <none>
I20250810 13:58:02.423993 21129 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:02.424248 21129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:02.424829 21129 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:02.429311 21129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9f28712fc401425490c21a42eb05d59e"
format_stamp: "Formatted at 2025-08-10 13:58:02 on dist-test-slave-k4vt"
I20250810 13:58:02.430474 21129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9f28712fc401425490c21a42eb05d59e"
format_stamp: "Formatted at 2025-08-10 13:58:02 on dist-test-slave-k4vt"
I20250810 13:58:02.438352 21129 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.009s
I20250810 13:58:02.444303 21145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:02.445581 21129 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250810 13:58:02.445959 21129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9f28712fc401425490c21a42eb05d59e"
format_stamp: "Formatted at 2025-08-10 13:58:02 on dist-test-slave-k4vt"
I20250810 13:58:02.446308 21129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:02.503134 21129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:02.504670 21129 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:02.505111 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:02.508242 21129 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:02.512573 21129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:02.512800 21129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:02.513089 21129 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:02.513299 21129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:02.691000 21129 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:33853
I20250810 13:58:02.691148 21257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:33853 every 8 connection(s)
I20250810 13:58:02.693861 21129 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:58:02.698060 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21129
I20250810 13:58:02.698556 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:58:02.706333 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:43769
--builtin_ntp_servers=127.19.27.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:02.739341 21258 heartbeater.cc:344] Connected to a master server at 127.19.27.126:43769
I20250810 13:58:02.739925 21258 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:02.741366 21258 heartbeater.cc:507] Master 127.19.27.126:43769 requested a full tablet report, sending...
I20250810 13:58:02.744741 21070 ts_manager.cc:194] Registered new tserver with Master: 9f28712fc401425490c21a42eb05d59e (127.19.27.65:33853)
I20250810 13:58:02.747637 21070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:57403
W20250810 13:58:03.019163 21262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:03.019704 21262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:03.020252 21262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:03.067824 21262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:03.068722 21262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:58:03.103469 21262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:03.104831 21262 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:03.106503 21262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:03.118319 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:03.751930 21258 heartbeater.cc:499] Master 127.19.27.126:43769 was elected leader, sending a full tablet report...
W20250810 13:58:03.123149 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:03.119828 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:04.309993 21270 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 13:58:04.310117 21262 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:04.314235 21262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:04.317036 21262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:04.318521 21262 hybrid_clock.cc:648] HybridClock initialized: now 1754834284318490 us; error 71 us; skew 500 ppm
I20250810 13:58:04.319309 21262 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:04.326375 21262 webserver.cc:489] Webserver started at http://127.19.27.66:44277/ using document root <none> and password file <none>
I20250810 13:58:04.327332 21262 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:04.327546 21262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:04.328048 21262 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:04.332639 21262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c932cd9690ed4108bbc085d4a8e2002b"
format_stamp: "Formatted at 2025-08-10 13:58:04 on dist-test-slave-k4vt"
I20250810 13:58:04.333730 21262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c932cd9690ed4108bbc085d4a8e2002b"
format_stamp: "Formatted at 2025-08-10 13:58:04 on dist-test-slave-k4vt"
I20250810 13:58:04.340947 21262 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250810 13:58:04.346694 21279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:04.347749 21262 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250810 13:58:04.348136 21262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c932cd9690ed4108bbc085d4a8e2002b"
format_stamp: "Formatted at 2025-08-10 13:58:04 on dist-test-slave-k4vt"
I20250810 13:58:04.348500 21262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:04.402199 21262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:04.403671 21262 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:04.404135 21262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:04.406667 21262 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:04.410889 21262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:04.411105 21262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:04.411373 21262 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:04.411518 21262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:04.546432 21262 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:41129
I20250810 13:58:04.546527 21391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:41129 every 8 connection(s)
I20250810 13:58:04.549052 21262 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:58:04.550933 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21262
I20250810 13:58:04.551738 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 13:58:04.559729 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:43769
--builtin_ntp_servers=127.19.27.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:04.571244 21392 heartbeater.cc:344] Connected to a master server at 127.19.27.126:43769
I20250810 13:58:04.571688 21392 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:04.572788 21392 heartbeater.cc:507] Master 127.19.27.126:43769 requested a full tablet report, sending...
I20250810 13:58:04.575064 21070 ts_manager.cc:194] Registered new tserver with Master: c932cd9690ed4108bbc085d4a8e2002b (127.19.27.66:41129)
I20250810 13:58:04.576426 21070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:56035
W20250810 13:58:04.862625 21396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:04.863170 21396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:04.863682 21396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:04.895037 21396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:04.895953 21396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:58:04.930593 21396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:04.931991 21396 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:04.933580 21396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:04.945597 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:05.579804 21392 heartbeater.cc:499] Master 127.19.27.126:43769 was elected leader, sending a full tablet report...
W20250810 13:58:04.946367 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:06.180521 21404 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1232 milliseconds
W20250810 13:58:06.181563 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:06.185042 21396 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.239s user 0.366s sys 0.872s
W20250810 13:58:06.185346 21396 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.240s user 0.366s sys 0.872s
I20250810 13:58:06.185564 21396 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:06.186615 21396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:06.188952 21396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:06.190330 21396 hybrid_clock.cc:648] HybridClock initialized: now 1754834286190300 us; error 48 us; skew 500 ppm
I20250810 13:58:06.191146 21396 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:06.198603 21396 webserver.cc:489] Webserver started at http://127.19.27.67:45429/ using document root <none> and password file <none>
I20250810 13:58:06.199684 21396 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:06.199990 21396 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:06.200466 21396 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:06.204952 21396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2eb368d8942f4e2180c97754b9e0a9fc"
format_stamp: "Formatted at 2025-08-10 13:58:06 on dist-test-slave-k4vt"
I20250810 13:58:06.206061 21396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2eb368d8942f4e2180c97754b9e0a9fc"
format_stamp: "Formatted at 2025-08-10 13:58:06 on dist-test-slave-k4vt"
I20250810 13:58:06.214185 21396 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 13:58:06.220631 21412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:06.221871 21396 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250810 13:58:06.222208 21396 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2eb368d8942f4e2180c97754b9e0a9fc"
format_stamp: "Formatted at 2025-08-10 13:58:06 on dist-test-slave-k4vt"
I20250810 13:58:06.222568 21396 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:06.295099 21396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:06.296697 21396 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:06.297151 21396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:06.299643 21396 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:06.304019 21396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:06.304262 21396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:06.304543 21396 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:06.304705 21396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:06.437621 21396 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:37831
I20250810 13:58:06.437747 21524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:37831 every 8 connection(s)
I20250810 13:58:06.440259 21396 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:58:06.445757 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21396
I20250810 13:58:06.446375 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 13:58:06.461968 21525 heartbeater.cc:344] Connected to a master server at 127.19.27.126:43769
I20250810 13:58:06.462455 21525 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:06.463474 21525 heartbeater.cc:507] Master 127.19.27.126:43769 requested a full tablet report, sending...
I20250810 13:58:06.465830 21070 ts_manager.cc:194] Registered new tserver with Master: 2eb368d8942f4e2180c97754b9e0a9fc (127.19.27.67:37831)
I20250810 13:58:06.467286 21070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:52393
I20250810 13:58:06.467567 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:06.503903 21070 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38812:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250810 13:58:06.523838 21070 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 13:58:06.582947 21460 tablet_service.cc:1468] Processing CreateTablet for tablet 6d00ae9636e94cfcbdb6561034cd392d (DEFAULT_TABLE table=TestTable [id=a51c37c13be54f17a9f4e5177f4b2d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:06.584682 21460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d00ae9636e94cfcbdb6561034cd392d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:06.585443 21327 tablet_service.cc:1468] Processing CreateTablet for tablet 6d00ae9636e94cfcbdb6561034cd392d (DEFAULT_TABLE table=TestTable [id=a51c37c13be54f17a9f4e5177f4b2d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:06.587371 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d00ae9636e94cfcbdb6561034cd392d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:06.595218 21193 tablet_service.cc:1468] Processing CreateTablet for tablet 6d00ae9636e94cfcbdb6561034cd392d (DEFAULT_TABLE table=TestTable [id=a51c37c13be54f17a9f4e5177f4b2d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:06.597425 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d00ae9636e94cfcbdb6561034cd392d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:06.610074 21544 tablet_bootstrap.cc:492] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: Bootstrap starting.
I20250810 13:58:06.615206 21545 tablet_bootstrap.cc:492] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: Bootstrap starting.
I20250810 13:58:06.617288 21544 tablet_bootstrap.cc:654] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:06.619768 21544 log.cc:826] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:06.622025 21545 tablet_bootstrap.cc:654] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:06.624655 21545 log.cc:826] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:06.625991 21544 tablet_bootstrap.cc:492] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: No bootstrap required, opened a new log
I20250810 13:58:06.626574 21544 ts_tablet_manager.cc:1397] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250810 13:58:06.626760 21547 tablet_bootstrap.cc:492] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: Bootstrap starting.
I20250810 13:58:06.630798 21545 tablet_bootstrap.cc:492] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: No bootstrap required, opened a new log
I20250810 13:58:06.631387 21545 ts_tablet_manager.cc:1397] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20250810 13:58:06.634039 21547 tablet_bootstrap.cc:654] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:06.635856 21547 log.cc:826] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:06.640759 21547 tablet_bootstrap.cc:492] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: No bootstrap required, opened a new log
I20250810 13:58:06.641214 21547 ts_tablet_manager.cc:1397] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20250810 13:58:06.653615 21544 raft_consensus.cc:357] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.654608 21544 raft_consensus.cc:383] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:06.654908 21544 raft_consensus.cc:738] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb368d8942f4e2180c97754b9e0a9fc, State: Initialized, Role: FOLLOWER
I20250810 13:58:06.655889 21544 consensus_queue.cc:260] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.659162 21525 heartbeater.cc:499] Master 127.19.27.126:43769 was elected leader, sending a full tablet report...
I20250810 13:58:06.658535 21545 raft_consensus.cc:357] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.659514 21545 raft_consensus.cc:383] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:06.659898 21545 raft_consensus.cc:738] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c932cd9690ed4108bbc085d4a8e2002b, State: Initialized, Role: FOLLOWER
I20250810 13:58:06.660655 21544 ts_tablet_manager.cc:1428] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: Time spent starting tablet: real 0.034s user 0.031s sys 0.001s
I20250810 13:58:06.660992 21545 consensus_queue.cc:260] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.662560 21547 raft_consensus.cc:357] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.663537 21547 raft_consensus.cc:383] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:06.663949 21547 raft_consensus.cc:738] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f28712fc401425490c21a42eb05d59e, State: Initialized, Role: FOLLOWER
I20250810 13:58:06.665546 21545 ts_tablet_manager.cc:1428] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: Time spent starting tablet: real 0.034s user 0.029s sys 0.004s
I20250810 13:58:06.666142 21547 consensus_queue.cc:260] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.670104 21547 ts_tablet_manager.cc:1428] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: Time spent starting tablet: real 0.029s user 0.009s sys 0.017s
W20250810 13:58:06.694809 21526 tablet.cc:2378] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 13:58:06.703315 21259 tablet.cc:2378] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 13:58:06.805994 21393 tablet.cc:2378] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:58:06.874570 21551 raft_consensus.cc:491] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:58:06.875085 21551 raft_consensus.cc:513] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.877599 21551 leader_election.cc:290] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f28712fc401425490c21a42eb05d59e (127.19.27.65:33853), 2eb368d8942f4e2180c97754b9e0a9fc (127.19.27.67:37831)
I20250810 13:58:06.888101 21552 raft_consensus.cc:491] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:58:06.888757 21552 raft_consensus.cc:513] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.889580 21213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d00ae9636e94cfcbdb6561034cd392d" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f28712fc401425490c21a42eb05d59e" is_pre_election: true
I20250810 13:58:06.889982 21480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d00ae9636e94cfcbdb6561034cd392d" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" is_pre_election: true
I20250810 13:58:06.890370 21213 raft_consensus.cc:2466] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 0.
I20250810 13:58:06.890698 21480 raft_consensus.cc:2466] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 0.
I20250810 13:58:06.891825 21282 leader_election.cc:304] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f28712fc401425490c21a42eb05d59e, c932cd9690ed4108bbc085d4a8e2002b; no voters:
I20250810 13:58:06.892061 21552 leader_election.cc:290] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c932cd9690ed4108bbc085d4a8e2002b (127.19.27.66:41129), 2eb368d8942f4e2180c97754b9e0a9fc (127.19.27.67:37831)
I20250810 13:58:06.893050 21551 raft_consensus.cc:2802] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 13:58:06.893419 21551 raft_consensus.cc:491] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:58:06.893726 21551 raft_consensus.cc:3058] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:06.901088 21551 raft_consensus.cc:513] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.903581 21213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d00ae9636e94cfcbdb6561034cd392d" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f28712fc401425490c21a42eb05d59e"
I20250810 13:58:06.903707 21480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d00ae9636e94cfcbdb6561034cd392d" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb368d8942f4e2180c97754b9e0a9fc"
I20250810 13:58:06.904232 21213 raft_consensus.cc:3058] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:06.904248 21480 raft_consensus.cc:3058] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:06.907204 21551 leader_election.cc:290] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 election: Requested vote from peers 9f28712fc401425490c21a42eb05d59e (127.19.27.65:33853), 2eb368d8942f4e2180c97754b9e0a9fc (127.19.27.67:37831)
I20250810 13:58:06.908016 21479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d00ae9636e94cfcbdb6561034cd392d" candidate_uuid: "9f28712fc401425490c21a42eb05d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" is_pre_election: true
I20250810 13:58:06.909272 21347 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d00ae9636e94cfcbdb6561034cd392d" candidate_uuid: "9f28712fc401425490c21a42eb05d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c932cd9690ed4108bbc085d4a8e2002b" is_pre_election: true
I20250810 13:58:06.910295 21347 raft_consensus.cc:2391] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f28712fc401425490c21a42eb05d59e in current term 1: Already voted for candidate c932cd9690ed4108bbc085d4a8e2002b in this term.
I20250810 13:58:06.911445 21480 raft_consensus.cc:2466] T 6d00ae9636e94cfcbdb6561034cd392d P 2eb368d8942f4e2180c97754b9e0a9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 1.
I20250810 13:58:06.911422 21148 leader_election.cc:304] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f28712fc401425490c21a42eb05d59e; no voters: 2eb368d8942f4e2180c97754b9e0a9fc, c932cd9690ed4108bbc085d4a8e2002b
I20250810 13:58:06.911444 21213 raft_consensus.cc:2466] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 1.
I20250810 13:58:06.912242 21552 raft_consensus.cc:2747] T 6d00ae9636e94cfcbdb6561034cd392d P 9f28712fc401425490c21a42eb05d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250810 13:58:06.912636 21281 leader_election.cc:304] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb368d8942f4e2180c97754b9e0a9fc, c932cd9690ed4108bbc085d4a8e2002b; no voters:
I20250810 13:58:06.913363 21551 raft_consensus.cc:2802] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:06.914837 21551 raft_consensus.cc:695] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [term 1 LEADER]: Becoming Leader. State: Replica: c932cd9690ed4108bbc085d4a8e2002b, State: Running, Role: LEADER
I20250810 13:58:06.915680 21551 consensus_queue.cc:237] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:06.927287 21068 catalog_manager.cc:5582] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b reported cstate change: term changed from 0 to 1, leader changed from <none> to c932cd9690ed4108bbc085d4a8e2002b (127.19.27.66). New cstate: current_term: 1 leader_uuid: "c932cd9690ed4108bbc085d4a8e2002b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } health_report { overall_health: UNKNOWN } } }
I20250810 13:58:06.953775 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:06.957402 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9f28712fc401425490c21a42eb05d59e to finish bootstrapping
I20250810 13:58:06.970008 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c932cd9690ed4108bbc085d4a8e2002b to finish bootstrapping
I20250810 13:58:06.981009 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2eb368d8942f4e2180c97754b9e0a9fc to finish bootstrapping
I20250810 13:58:06.994779 21068 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38812:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250810 13:58:06.996364 21068 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 13:58:07.013085 21193 tablet_service.cc:1468] Processing CreateTablet for tablet 7b3d1a75c93442b1bfe6eca91c07bdb4 (DEFAULT_TABLE table=TestAnotherTable [id=ebce4fb3316f4845bf8a19331de0d7f6]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250810 13:58:07.013650 21327 tablet_service.cc:1468] Processing CreateTablet for tablet 7b3d1a75c93442b1bfe6eca91c07bdb4 (DEFAULT_TABLE table=TestAnotherTable [id=ebce4fb3316f4845bf8a19331de0d7f6]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250810 13:58:07.014235 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3d1a75c93442b1bfe6eca91c07bdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:07.014403 21460 tablet_service.cc:1468] Processing CreateTablet for tablet 7b3d1a75c93442b1bfe6eca91c07bdb4 (DEFAULT_TABLE table=TestAnotherTable [id=ebce4fb3316f4845bf8a19331de0d7f6]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250810 13:58:07.014698 21327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3d1a75c93442b1bfe6eca91c07bdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:07.015280 21460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3d1a75c93442b1bfe6eca91c07bdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:07.026983 21544 tablet_bootstrap.cc:492] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc: Bootstrap starting.
I20250810 13:58:07.029363 21547 tablet_bootstrap.cc:492] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e: Bootstrap starting.
I20250810 13:58:07.030148 21545 tablet_bootstrap.cc:492] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b: Bootstrap starting.
I20250810 13:58:07.032771 21544 tablet_bootstrap.cc:654] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:07.035808 21547 tablet_bootstrap.cc:654] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:07.036594 21545 tablet_bootstrap.cc:654] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:07.045281 21547 tablet_bootstrap.cc:492] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e: No bootstrap required, opened a new log
I20250810 13:58:07.045281 21544 tablet_bootstrap.cc:492] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc: No bootstrap required, opened a new log
I20250810 13:58:07.045424 21545 tablet_bootstrap.cc:492] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b: No bootstrap required, opened a new log
I20250810 13:58:07.045714 21544 ts_tablet_manager.cc:1397] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20250810 13:58:07.045810 21547 ts_tablet_manager.cc:1397] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.002s
I20250810 13:58:07.045852 21545 ts_tablet_manager.cc:1397] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250810 13:58:07.048413 21545 raft_consensus.cc:357] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.048418 21547 raft_consensus.cc:357] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.048417 21544 raft_consensus.cc:357] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.049083 21545 raft_consensus.cc:383] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:07.049122 21547 raft_consensus.cc:383] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:07.049117 21544 raft_consensus.cc:383] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:07.049345 21545 raft_consensus.cc:738] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c932cd9690ed4108bbc085d4a8e2002b, State: Initialized, Role: FOLLOWER
I20250810 13:58:07.049400 21547 raft_consensus.cc:738] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f28712fc401425490c21a42eb05d59e, State: Initialized, Role: FOLLOWER
I20250810 13:58:07.049399 21544 raft_consensus.cc:738] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb368d8942f4e2180c97754b9e0a9fc, State: Initialized, Role: FOLLOWER
I20250810 13:58:07.050073 21544 consensus_queue.cc:260] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.050068 21547 consensus_queue.cc:260] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.052575 21544 ts_tablet_manager.cc:1428] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250810 13:58:07.054502 21547 ts_tablet_manager.cc:1428] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250810 13:58:07.050518 21545 consensus_queue.cc:260] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.056694 21545 ts_tablet_manager.cc:1428] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250810 13:58:07.127370 21551 raft_consensus.cc:491] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:58:07.127882 21551 raft_consensus.cc:513] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.129484 21551 leader_election.cc:290] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f28712fc401425490c21a42eb05d59e (127.19.27.65:33853), 2eb368d8942f4e2180c97754b9e0a9fc (127.19.27.67:37831)
I20250810 13:58:07.130322 21213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b3d1a75c93442b1bfe6eca91c07bdb4" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f28712fc401425490c21a42eb05d59e" is_pre_election: true
I20250810 13:58:07.130514 21480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b3d1a75c93442b1bfe6eca91c07bdb4" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" is_pre_election: true
I20250810 13:58:07.130800 21213 raft_consensus.cc:2466] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 0.
I20250810 13:58:07.131049 21480 raft_consensus.cc:2466] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 0.
I20250810 13:58:07.131631 21282 leader_election.cc:304] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f28712fc401425490c21a42eb05d59e, c932cd9690ed4108bbc085d4a8e2002b; no voters:
I20250810 13:58:07.132339 21551 raft_consensus.cc:2802] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 13:58:07.132665 21551 raft_consensus.cc:491] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:58:07.132968 21551 raft_consensus.cc:3058] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:07.137373 21551 raft_consensus.cc:513] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.138754 21551 leader_election.cc:290] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 election: Requested vote from peers 9f28712fc401425490c21a42eb05d59e (127.19.27.65:33853), 2eb368d8942f4e2180c97754b9e0a9fc (127.19.27.67:37831)
I20250810 13:58:07.139722 21213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b3d1a75c93442b1bfe6eca91c07bdb4" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f28712fc401425490c21a42eb05d59e"
I20250810 13:58:07.139889 21480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b3d1a75c93442b1bfe6eca91c07bdb4" candidate_uuid: "c932cd9690ed4108bbc085d4a8e2002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb368d8942f4e2180c97754b9e0a9fc"
I20250810 13:58:07.140221 21213 raft_consensus.cc:3058] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:07.140347 21480 raft_consensus.cc:3058] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:07.144140 21213 raft_consensus.cc:2466] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 9f28712fc401425490c21a42eb05d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 1.
I20250810 13:58:07.144342 21480 raft_consensus.cc:2466] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P 2eb368d8942f4e2180c97754b9e0a9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c932cd9690ed4108bbc085d4a8e2002b in term 1.
I20250810 13:58:07.145186 21282 leader_election.cc:304] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f28712fc401425490c21a42eb05d59e, c932cd9690ed4108bbc085d4a8e2002b; no voters:
I20250810 13:58:07.145793 21551 raft_consensus.cc:2802] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:07.146175 21551 raft_consensus.cc:695] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [term 1 LEADER]: Becoming Leader. State: Replica: c932cd9690ed4108bbc085d4a8e2002b, State: Running, Role: LEADER
I20250810 13:58:07.146746 21551 consensus_queue.cc:237] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } }
I20250810 13:58:07.153421 21069 catalog_manager.cc:5582] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b reported cstate change: term changed from 0 to 1, leader changed from <none> to c932cd9690ed4108bbc085d4a8e2002b (127.19.27.66). New cstate: current_term: 1 leader_uuid: "c932cd9690ed4108bbc085d4a8e2002b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c932cd9690ed4108bbc085d4a8e2002b" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 41129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 } health_report { overall_health: UNKNOWN } } }
I20250810 13:58:07.389657 21551 consensus_queue.cc:1035] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:58:07.409675 21551 consensus_queue.cc:1035] T 6d00ae9636e94cfcbdb6561034cd392d P c932cd9690ed4108bbc085d4a8e2002b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250810 13:58:07.514751 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:07.515364 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:07.550861 21564 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250810 13:58:07.596158 21551 consensus_queue.cc:1035] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f28712fc401425490c21a42eb05d59e" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:58:07.614352 21551 consensus_queue.cc:1035] T 7b3d1a75c93442b1bfe6eca91c07bdb4 P c932cd9690ed4108bbc085d4a8e2002b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb368d8942f4e2180c97754b9e0a9fc" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250810 13:58:08.951352 21564 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.316s user 0.000s sys 0.002s
W20250810 13:58:08.951674 21564 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.316s user 0.000s sys 0.002s
W20250810 13:58:10.350859 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:10.351472 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:10.383319 21586 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 13:58:11.678326 21586 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.254s user 0.519s sys 0.734s
W20250810 13:58:11.678632 21586 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.255s user 0.519s sys 0.734s
W20250810 13:58:13.065742 21601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:13.066345 21601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:13.099221 21601 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 13:58:14.407862 21601 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.268s user 0.471s sys 0.795s
W20250810 13:58:14.408192 21601 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.269s user 0.474s sys 0.795s
W20250810 13:58:15.814504 21617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:15.815141 21617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:15.846357 21617 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 13:58:17.190099 21617 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.303s user 0.487s sys 0.809s
W20250810 13:58:17.190524 21617 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.304s user 0.487s sys 0.809s
I20250810 13:58:18.275095 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21129
I20250810 13:58:18.302211 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21262
I20250810 13:58:18.330191 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21396
I20250810 13:58:18.355304 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21037
2025-08-10T13:58:18Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (20222 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250810 13:58:18.415067 19565 test_util.cc:276] Using random seed: 855582656
I20250810 13:58:18.419354 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:58:18.419530 19565 ts_itest-base.cc:116] --------------
I20250810 13:58:18.419704 19565 ts_itest-base.cc:117] 3 tablet servers
I20250810 13:58:18.419893 19565 ts_itest-base.cc:118] 3 replicas per TS
I20250810 13:58:18.420110 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:58:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:58:18Z Disabled control of system clock
I20250810 13:58:18.469951 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:37285
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:37285
--superuser_acl=no-such-user with env {}
W20250810 13:58:18.777379 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:18.777940 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:18.778383 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:18.810089 21638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:58:18.810412 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:18.810626 21638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:58:18.810838 21638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:58:18.846416 21638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:37285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:18.847643 21638 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:18.849287 21638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:18.861158 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:18.861285 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:20.166832 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:20.170997 21646 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1310 milliseconds
W20250810 13:58:20.172029 21638 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.311s user 0.420s sys 0.883s
W20250810 13:58:20.172309 21638 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.312s user 0.420s sys 0.883s
I20250810 13:58:20.172528 21638 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:20.173616 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:20.176318 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:20.177695 21638 hybrid_clock.cc:648] HybridClock initialized: now 1754834300177648 us; error 62 us; skew 500 ppm
I20250810 13:58:20.178515 21638 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:20.186995 21638 webserver.cc:489] Webserver started at http://127.19.27.126:41139/ using document root <none> and password file <none>
I20250810 13:58:20.187966 21638 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:20.188194 21638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:20.188678 21638 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:20.193121 21638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "48e80b8400494320be1d5ea7d629afd3"
format_stamp: "Formatted at 2025-08-10 13:58:20 on dist-test-slave-k4vt"
I20250810 13:58:20.194242 21638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "48e80b8400494320be1d5ea7d629afd3"
format_stamp: "Formatted at 2025-08-10 13:58:20 on dist-test-slave-k4vt"
I20250810 13:58:20.202445 21638 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.005s
I20250810 13:58:20.209262 21654 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:20.210436 21638 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250810 13:58:20.210804 21638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "48e80b8400494320be1d5ea7d629afd3"
format_stamp: "Formatted at 2025-08-10 13:58:20 on dist-test-slave-k4vt"
I20250810 13:58:20.211131 21638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:20.277726 21638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:20.279207 21638 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:20.279709 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:20.349853 21638 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:37285
I20250810 13:58:20.349924 21705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:37285 every 8 connection(s)
I20250810 13:58:20.352667 21638 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:58:20.357177 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21638
I20250810 13:58:20.357738 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:58:20.358117 21706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:20.381965 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3: Bootstrap starting.
I20250810 13:58:20.388186 21706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:20.389954 21706 log.cc:826] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:20.394562 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3: No bootstrap required, opened a new log
I20250810 13:58:20.411691 21706 raft_consensus.cc:357] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e80b8400494320be1d5ea7d629afd3" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 37285 } }
I20250810 13:58:20.412441 21706 raft_consensus.cc:383] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:20.412688 21706 raft_consensus.cc:738] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e80b8400494320be1d5ea7d629afd3, State: Initialized, Role: FOLLOWER
I20250810 13:58:20.413410 21706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e80b8400494320be1d5ea7d629afd3" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 37285 } }
I20250810 13:58:20.413905 21706 raft_consensus.cc:397] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:20.414165 21706 raft_consensus.cc:491] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:20.414494 21706 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:20.418620 21706 raft_consensus.cc:513] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e80b8400494320be1d5ea7d629afd3" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 37285 } }
I20250810 13:58:20.419410 21706 leader_election.cc:304] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e80b8400494320be1d5ea7d629afd3; no voters:
I20250810 13:58:20.421161 21706 leader_election.cc:290] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:58:20.421921 21711 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:20.424116 21711 raft_consensus.cc:695] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [term 1 LEADER]: Becoming Leader. State: Replica: 48e80b8400494320be1d5ea7d629afd3, State: Running, Role: LEADER
I20250810 13:58:20.424770 21711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e80b8400494320be1d5ea7d629afd3" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 37285 } }
I20250810 13:58:20.425267 21706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:58:20.433696 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e80b8400494320be1d5ea7d629afd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e80b8400494320be1d5ea7d629afd3" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 37285 } } }
I20250810 13:58:20.434516 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:20.435266 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e80b8400494320be1d5ea7d629afd3. Latest consensus state: current_term: 1 leader_uuid: "48e80b8400494320be1d5ea7d629afd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e80b8400494320be1d5ea7d629afd3" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 37285 } } }
I20250810 13:58:20.436538 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:20.439252 21720 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:58:20.450223 21720 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:58:20.468086 21720 catalog_manager.cc:1349] Generated new cluster ID: a3aa114d82af473bb4b89f93266836be
I20250810 13:58:20.468405 21720 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:58:20.484295 21720 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:58:20.485881 21720 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:58:20.506251 21720 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 48e80b8400494320be1d5ea7d629afd3: Generated new TSK 0
I20250810 13:58:20.507453 21720 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:58:20.521868 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:37285
--builtin_ntp_servers=127.19.27.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 13:58:20.836472 21730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:20.836979 21730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:20.837503 21730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:20.870226 21730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:20.871115 21730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:58:20.907519 21730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:37285
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:20.908946 21730 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:20.910624 21730 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:20.923882 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:20.929929 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:20.925596 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:22.381541 21738 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1454 milliseconds
I20250810 13:58:22.381698 21730 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:22.383284 21730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:22.386061 21730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:22.387583 21730 hybrid_clock.cc:648] HybridClock initialized: now 1754834302387519 us; error 58 us; skew 500 ppm
I20250810 13:58:22.388720 21730 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:22.396484 21730 webserver.cc:489] Webserver started at http://127.19.27.65:41071/ using document root <none> and password file <none>
I20250810 13:58:22.397874 21730 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:22.398157 21730 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:22.398805 21730 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:22.405371 21730 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "603545fda79c45ed9f1929d15558a968"
format_stamp: "Formatted at 2025-08-10 13:58:22 on dist-test-slave-k4vt"
I20250810 13:58:22.406906 21730 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "603545fda79c45ed9f1929d15558a968"
format_stamp: "Formatted at 2025-08-10 13:58:22 on dist-test-slave-k4vt"
I20250810 13:58:22.416451 21730 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250810 13:58:22.424156 21746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:22.425351 21730 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250810 13:58:22.425770 21730 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "603545fda79c45ed9f1929d15558a968"
format_stamp: "Formatted at 2025-08-10 13:58:22 on dist-test-slave-k4vt"
I20250810 13:58:22.426213 21730 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:22.481190 21730 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:22.482731 21730 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:22.483218 21730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:22.486322 21730 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:22.491194 21730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:22.491421 21730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:22.491674 21730 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:22.491864 21730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250810 13:58:22.663727 21730 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:41483
I20250810 13:58:22.663933 21858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:41483 every 8 connection(s)
I20250810 13:58:22.666406 21730 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:58:22.670130 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21730
I20250810 13:58:22.670750 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:58:22.684428 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:37285
--builtin_ntp_servers=127.19.27.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:22.699283 21859 heartbeater.cc:344] Connected to a master server at 127.19.27.126:37285
I20250810 13:58:22.699719 21859 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:22.700824 21859 heartbeater.cc:507] Master 127.19.27.126:37285 requested a full tablet report, sending...
I20250810 13:58:22.703665 21671 ts_manager.cc:194] Registered new tserver with Master: 603545fda79c45ed9f1929d15558a968 (127.19.27.65:41483)
I20250810 13:58:22.705679 21671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:43401
W20250810 13:58:23.080164 21863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:23.080716 21863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:23.081233 21863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:23.113237 21863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:23.114107 21863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:58:23.150449 21863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:37285
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:23.151997 21863 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:23.153702 21863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:23.166234 21869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:23.709266 21859 heartbeater.cc:499] Master 127.19.27.126:37285 was elected leader, sending a full tablet report...
W20250810 13:58:24.569774 21868 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 21863
W20250810 13:58:24.618044 21868 kernel_stack_watchdog.cc:198] Thread 21863 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:58:23.166806 21870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:24.618687 21863 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.452s user 0.450s sys 0.993s
W20250810 13:58:24.619048 21863 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.453s user 0.450s sys 0.993s
W20250810 13:58:24.621990 21872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:24.624191 21871 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1452 milliseconds
I20250810 13:58:24.624217 21863 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:24.625523 21863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:24.627630 21863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:24.628995 21863 hybrid_clock.cc:648] HybridClock initialized: now 1754834304628969 us; error 43 us; skew 500 ppm
I20250810 13:58:24.629818 21863 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:24.635998 21863 webserver.cc:489] Webserver started at http://127.19.27.66:41909/ using document root <none> and password file <none>
I20250810 13:58:24.636951 21863 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:24.637143 21863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:24.637620 21863 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:24.642216 21863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2a54211ca2fd4f5e8c81db92aa78bbda"
format_stamp: "Formatted at 2025-08-10 13:58:24 on dist-test-slave-k4vt"
I20250810 13:58:24.643417 21863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2a54211ca2fd4f5e8c81db92aa78bbda"
format_stamp: "Formatted at 2025-08-10 13:58:24 on dist-test-slave-k4vt"
I20250810 13:58:24.651026 21863 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250810 13:58:24.657056 21880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:24.658229 21863 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250810 13:58:24.658636 21863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2a54211ca2fd4f5e8c81db92aa78bbda"
format_stamp: "Formatted at 2025-08-10 13:58:24 on dist-test-slave-k4vt"
I20250810 13:58:24.659015 21863 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:24.722931 21863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:24.724527 21863 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:24.724990 21863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:24.727483 21863 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:24.731565 21863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:24.731796 21863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:24.732038 21863 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:24.732192 21863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:24.864962 21863 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:34849
I20250810 13:58:24.865068 21992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:34849 every 8 connection(s)
I20250810 13:58:24.867630 21863 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:58:24.868321 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21863
I20250810 13:58:24.868841 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 13:58:24.877101 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:37285
--builtin_ntp_servers=127.19.27.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:24.893682 21993 heartbeater.cc:344] Connected to a master server at 127.19.27.126:37285
I20250810 13:58:24.894147 21993 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:24.895246 21993 heartbeater.cc:507] Master 127.19.27.126:37285 requested a full tablet report, sending...
I20250810 13:58:24.897508 21671 ts_manager.cc:194] Registered new tserver with Master: 2a54211ca2fd4f5e8c81db92aa78bbda (127.19.27.66:34849)
I20250810 13:58:24.898736 21671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:47157
W20250810 13:58:25.182869 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:25.183409 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:25.183969 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:25.216953 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:25.217792 21997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:58:25.252641 21997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:37285
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:25.254123 21997 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:25.255726 21997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:25.268959 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:25.902143 21993 heartbeater.cc:499] Master 127.19.27.126:37285 was elected leader, sending a full tablet report...
W20250810 13:58:25.269333 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:26.529850 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:26.532104 21997 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.264s user 0.370s sys 0.887s
W20250810 13:58:26.532500 21997 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.264s user 0.370s sys 0.887s
W20250810 13:58:26.533252 22005 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1264 milliseconds
I20250810 13:58:26.533310 21997 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:26.534981 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:26.537868 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:26.539520 21997 hybrid_clock.cc:648] HybridClock initialized: now 1754834306539459 us; error 50 us; skew 500 ppm
I20250810 13:58:26.540881 21997 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:26.550621 21997 webserver.cc:489] Webserver started at http://127.19.27.67:41533/ using document root <none> and password file <none>
I20250810 13:58:26.552192 21997 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:26.552531 21997 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:26.553193 21997 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:26.560853 21997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fc278fc0191240df8fa5ea3a77e1fb8f"
format_stamp: "Formatted at 2025-08-10 13:58:26 on dist-test-slave-k4vt"
I20250810 13:58:26.562626 21997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fc278fc0191240df8fa5ea3a77e1fb8f"
format_stamp: "Formatted at 2025-08-10 13:58:26 on dist-test-slave-k4vt"
I20250810 13:58:26.573199 21997 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.010s sys 0.002s
I20250810 13:58:26.581629 22013 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:26.582911 21997 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250810 13:58:26.583366 21997 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fc278fc0191240df8fa5ea3a77e1fb8f"
format_stamp: "Formatted at 2025-08-10 13:58:26 on dist-test-slave-k4vt"
I20250810 13:58:26.583884 21997 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:26.676623 21997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:26.678170 21997 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:26.678639 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:26.681408 21997 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:26.685891 21997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:26.686124 21997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:26.686342 21997 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:26.686499 21997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:26.822599 21997 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:36447
I20250810 13:58:26.822655 22125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:36447 every 8 connection(s)
I20250810 13:58:26.825466 21997 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:58:26.826659 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 21997
I20250810 13:58:26.827029 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 13:58:26.850278 22126 heartbeater.cc:344] Connected to a master server at 127.19.27.126:37285
I20250810 13:58:26.850692 22126 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:26.851697 22126 heartbeater.cc:507] Master 127.19.27.126:37285 requested a full tablet report, sending...
I20250810 13:58:26.853730 21671 ts_manager.cc:194] Registered new tserver with Master: fc278fc0191240df8fa5ea3a77e1fb8f (127.19.27.67:36447)
I20250810 13:58:26.854910 21671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:40233
I20250810 13:58:26.861908 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:26.895825 21671 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50430:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250810 13:58:26.916010 21671 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 13:58:26.968533 21794 tablet_service.cc:1468] Processing CreateTablet for tablet 6483eabe93c94e17ae4f1d63b6f70e5b (DEFAULT_TABLE table=TestTable [id=4afb06588bdb4afbbcdb3aa686505a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:26.971825 21794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6483eabe93c94e17ae4f1d63b6f70e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:26.973735 22061 tablet_service.cc:1468] Processing CreateTablet for tablet 6483eabe93c94e17ae4f1d63b6f70e5b (DEFAULT_TABLE table=TestTable [id=4afb06588bdb4afbbcdb3aa686505a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:26.974992 21928 tablet_service.cc:1468] Processing CreateTablet for tablet 6483eabe93c94e17ae4f1d63b6f70e5b (DEFAULT_TABLE table=TestTable [id=4afb06588bdb4afbbcdb3aa686505a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:26.975706 22061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6483eabe93c94e17ae4f1d63b6f70e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:26.976490 21928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6483eabe93c94e17ae4f1d63b6f70e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:26.997529 22145 tablet_bootstrap.cc:492] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: Bootstrap starting.
I20250810 13:58:27.000836 22146 tablet_bootstrap.cc:492] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: Bootstrap starting.
I20250810 13:58:27.005031 22147 tablet_bootstrap.cc:492] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: Bootstrap starting.
I20250810 13:58:27.006491 22145 tablet_bootstrap.cc:654] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:27.008436 22145 log.cc:826] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:27.008709 22146 tablet_bootstrap.cc:654] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:27.011246 22146 log.cc:826] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:27.013212 22147 tablet_bootstrap.cc:654] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:27.013756 22145 tablet_bootstrap.cc:492] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: No bootstrap required, opened a new log
I20250810 13:58:27.014312 22145 ts_tablet_manager.cc:1397] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20250810 13:58:27.015524 22147 log.cc:826] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:27.017158 22146 tablet_bootstrap.cc:492] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: No bootstrap required, opened a new log
I20250810 13:58:27.017824 22146 ts_tablet_manager.cc:1397] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.002s
I20250810 13:58:27.022037 22147 tablet_bootstrap.cc:492] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: No bootstrap required, opened a new log
I20250810 13:58:27.022593 22147 ts_tablet_manager.cc:1397] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20250810 13:58:27.037566 22146 raft_consensus.cc:357] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.038303 22146 raft_consensus.cc:383] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:27.038573 22146 raft_consensus.cc:738] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603545fda79c45ed9f1929d15558a968, State: Initialized, Role: FOLLOWER
I20250810 13:58:27.039353 22146 consensus_queue.cc:260] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.041391 22145 raft_consensus.cc:357] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.042383 22145 raft_consensus.cc:383] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:27.042730 22145 raft_consensus.cc:738] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a54211ca2fd4f5e8c81db92aa78bbda, State: Initialized, Role: FOLLOWER
I20250810 13:58:27.043824 22146 ts_tablet_manager.cc:1428] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: Time spent starting tablet: real 0.026s user 0.020s sys 0.004s
I20250810 13:58:27.043838 22145 consensus_queue.cc:260] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.048327 22145 ts_tablet_manager.cc:1428] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: Time spent starting tablet: real 0.034s user 0.027s sys 0.009s
I20250810 13:58:27.051479 22147 raft_consensus.cc:357] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.052304 22147 raft_consensus.cc:383] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:27.052551 22147 raft_consensus.cc:738] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc278fc0191240df8fa5ea3a77e1fb8f, State: Initialized, Role: FOLLOWER
I20250810 13:58:27.053448 22147 consensus_queue.cc:260] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.056706 22126 heartbeater.cc:499] Master 127.19.27.126:37285 was elected leader, sending a full tablet report...
I20250810 13:58:27.058071 22147 ts_tablet_manager.cc:1428] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: Time spent starting tablet: real 0.035s user 0.035s sys 0.000s
W20250810 13:58:27.080431 22127 tablet.cc:2378] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 13:58:27.126147 21994 tablet.cc:2378] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:58:27.179117 22151 raft_consensus.cc:491] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:58:27.179612 22151 raft_consensus.cc:513] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.182109 22151 leader_election.cc:290] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc278fc0191240df8fa5ea3a77e1fb8f (127.19.27.67:36447), 2a54211ca2fd4f5e8c81db92aa78bbda (127.19.27.66:34849)
W20250810 13:58:27.183300 21860 tablet.cc:2378] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:58:27.194552 22081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6483eabe93c94e17ae4f1d63b6f70e5b" candidate_uuid: "603545fda79c45ed9f1929d15558a968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" is_pre_election: true
I20250810 13:58:27.195314 21948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6483eabe93c94e17ae4f1d63b6f70e5b" candidate_uuid: "603545fda79c45ed9f1929d15558a968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" is_pre_election: true
I20250810 13:58:27.195470 22081 raft_consensus.cc:2466] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603545fda79c45ed9f1929d15558a968 in term 0.
I20250810 13:58:27.196225 21948 raft_consensus.cc:2466] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603545fda79c45ed9f1929d15558a968 in term 0.
I20250810 13:58:27.196843 21748 leader_election.cc:304] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603545fda79c45ed9f1929d15558a968, fc278fc0191240df8fa5ea3a77e1fb8f; no voters:
I20250810 13:58:27.197703 22151 raft_consensus.cc:2802] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 13:58:27.198000 22151 raft_consensus.cc:491] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:58:27.198264 22151 raft_consensus.cc:3058] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:27.202742 22151 raft_consensus.cc:513] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.204147 22151 leader_election.cc:290] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [CANDIDATE]: Term 1 election: Requested vote from peers fc278fc0191240df8fa5ea3a77e1fb8f (127.19.27.67:36447), 2a54211ca2fd4f5e8c81db92aa78bbda (127.19.27.66:34849)
I20250810 13:58:27.204989 22081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6483eabe93c94e17ae4f1d63b6f70e5b" candidate_uuid: "603545fda79c45ed9f1929d15558a968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f"
I20250810 13:58:27.205008 21948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6483eabe93c94e17ae4f1d63b6f70e5b" candidate_uuid: "603545fda79c45ed9f1929d15558a968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda"
I20250810 13:58:27.205415 22081 raft_consensus.cc:3058] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:27.205447 21948 raft_consensus.cc:3058] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:27.209969 22081 raft_consensus.cc:2466] T 6483eabe93c94e17ae4f1d63b6f70e5b P fc278fc0191240df8fa5ea3a77e1fb8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603545fda79c45ed9f1929d15558a968 in term 1.
I20250810 13:58:27.209966 21948 raft_consensus.cc:2466] T 6483eabe93c94e17ae4f1d63b6f70e5b P 2a54211ca2fd4f5e8c81db92aa78bbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603545fda79c45ed9f1929d15558a968 in term 1.
I20250810 13:58:27.210884 21748 leader_election.cc:304] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603545fda79c45ed9f1929d15558a968, fc278fc0191240df8fa5ea3a77e1fb8f; no voters:
I20250810 13:58:27.211513 22151 raft_consensus.cc:2802] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:27.213035 22151 raft_consensus.cc:695] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [term 1 LEADER]: Becoming Leader. State: Replica: 603545fda79c45ed9f1929d15558a968, State: Running, Role: LEADER
I20250810 13:58:27.213845 22151 consensus_queue.cc:237] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } }
I20250810 13:58:27.224105 21671 catalog_manager.cc:5582] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603545fda79c45ed9f1929d15558a968 (127.19.27.65). New cstate: current_term: 1 leader_uuid: "603545fda79c45ed9f1929d15558a968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603545fda79c45ed9f1929d15558a968" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 } health_report { overall_health: UNKNOWN } } }
I20250810 13:58:27.264613 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:27.267622 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 603545fda79c45ed9f1929d15558a968 to finish bootstrapping
I20250810 13:58:27.279927 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2a54211ca2fd4f5e8c81db92aa78bbda to finish bootstrapping
I20250810 13:58:27.290935 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fc278fc0191240df8fa5ea3a77e1fb8f to finish bootstrapping
I20250810 13:58:27.654604 22151 consensus_queue.cc:1035] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc278fc0191240df8fa5ea3a77e1fb8f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 13:58:27.676538 22151 consensus_queue.cc:1035] T 6483eabe93c94e17ae4f1d63b6f70e5b P 603545fda79c45ed9f1929d15558a968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a54211ca2fd4f5e8c81db92aa78bbda" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250810 13:58:28.972481 21671 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:50450
W20250810 13:58:29.096438 22122 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac480 after lost signal to thread 21998
W20250810 13:58:29.097209 22122 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcec0 after lost signal to thread 22125
I20250810 13:58:30.006068 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21730
I20250810 13:58:30.031328 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21863
I20250810 13:58:30.055507 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21997
I20250810 13:58:30.076056 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 21638
2025-08-10T13:58:30Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11714 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250810 13:58:30.129071 19565 test_util.cc:276] Using random seed: 867296660
I20250810 13:58:30.132836 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:58:30.132987 19565 ts_itest-base.cc:116] --------------
I20250810 13:58:30.133147 19565 ts_itest-base.cc:117] 3 tablet servers
I20250810 13:58:30.133281 19565 ts_itest-base.cc:118] 3 replicas per TS
I20250810 13:58:30.133411 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:58:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:58:30Z Disabled control of system clock
I20250810 13:58:30.168165 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41133
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:41133 with env {}
W20250810 13:58:30.457127 22194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:30.457629 22194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:30.458042 22194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:30.487128 22194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:58:30.487401 22194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:30.487592 22194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:58:30.487788 22194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:58:30.521312 22194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:41133
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:30.522500 22194 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:30.524055 22194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:30.534020 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:31.938019 22199 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 22194
W20250810 13:58:32.243633 22199 kernel_stack_watchdog.cc:198] Thread 22194 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:58:32.244052 22194 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.710s user 0.651s sys 1.057s
W20250810 13:58:32.244966 22194 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.711s user 0.651s sys 1.058s
W20250810 13:58:30.534466 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:32.246758 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:32.250277 22202 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1712 milliseconds
I20250810 13:58:32.250291 22194 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:32.251474 22194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:32.253661 22194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:32.254983 22194 hybrid_clock.cc:648] HybridClock initialized: now 1754834312254922 us; error 60 us; skew 500 ppm
I20250810 13:58:32.255821 22194 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:32.263361 22194 webserver.cc:489] Webserver started at http://127.19.27.126:36591/ using document root <none> and password file <none>
I20250810 13:58:32.264351 22194 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:32.264539 22194 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:32.264990 22194 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:32.268872 22194 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2d627fde845446419ace1c1815205f13"
format_stamp: "Formatted at 2025-08-10 13:58:32 on dist-test-slave-k4vt"
I20250810 13:58:32.269811 22194 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2d627fde845446419ace1c1815205f13"
format_stamp: "Formatted at 2025-08-10 13:58:32 on dist-test-slave-k4vt"
I20250810 13:58:32.276249 22194 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250810 13:58:32.281204 22210 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:32.282228 22194 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250810 13:58:32.282523 22194 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2d627fde845446419ace1c1815205f13"
format_stamp: "Formatted at 2025-08-10 13:58:32 on dist-test-slave-k4vt"
I20250810 13:58:32.282824 22194 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:32.337801 22194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:32.339107 22194 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:32.339521 22194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:32.408485 22194 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:41133
I20250810 13:58:32.408555 22261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:41133 every 8 connection(s)
I20250810 13:58:32.410995 22194 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:58:32.415951 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 22194
I20250810 13:58:32.416047 22262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:32.416402 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:58:32.439088 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap starting.
I20250810 13:58:32.444814 22262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:32.446305 22262 log.cc:826] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:32.450328 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: No bootstrap required, opened a new log
I20250810 13:58:32.465902 22262 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:32.466480 22262 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:32.466701 22262 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d627fde845446419ace1c1815205f13, State: Initialized, Role: FOLLOWER
I20250810 13:58:32.467463 22262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:32.468027 22262 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:32.468292 22262 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:32.468591 22262 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:32.472325 22262 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:32.472947 22262 leader_election.cc:304] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d627fde845446419ace1c1815205f13; no voters:
I20250810 13:58:32.474603 22262 leader_election.cc:290] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:58:32.475373 22267 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:32.477440 22267 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 1 LEADER]: Becoming Leader. State: Replica: 2d627fde845446419ace1c1815205f13, State: Running, Role: LEADER
I20250810 13:58:32.478179 22267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:32.479104 22262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:58:32.488629 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:58:32.489392 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:32.491732 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d627fde845446419ace1c1815205f13. Latest consensus state: current_term: 1 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:58:32.492511 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:32.497716 22276 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:58:32.508415 22276 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:58:32.522436 22276 catalog_manager.cc:1349] Generated new cluster ID: d97a81b24ea84adeb0c23ddf0223b00c
I20250810 13:58:32.522646 22276 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:58:32.553130 22276 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:58:32.554663 22276 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:58:32.566597 22276 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Generated new TSK 0
I20250810 13:58:32.567271 22276 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:58:32.573899 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41133
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 13:58:32.868860 22286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:32.869290 22286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:32.869817 22286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:32.899147 22286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:32.899984 22286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:58:32.932421 22286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:32.933669 22286 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:32.935248 22286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:32.947245 22292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:32.948832 22293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:32.951578 22295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:34.467442 22294 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1517 milliseconds
I20250810 13:58:34.467557 22286 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:34.472395 22286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:34.475185 22286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:34.476670 22286 hybrid_clock.cc:648] HybridClock initialized: now 1754834314476623 us; error 72 us; skew 500 ppm
I20250810 13:58:34.477504 22286 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:34.484110 22286 webserver.cc:489] Webserver started at http://127.19.27.65:33051/ using document root <none> and password file <none>
I20250810 13:58:34.485126 22286 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:34.485369 22286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:34.485893 22286 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:34.490417 22286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4709b59128be4d74a9c65521d206ef95"
format_stamp: "Formatted at 2025-08-10 13:58:34 on dist-test-slave-k4vt"
I20250810 13:58:34.491590 22286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4709b59128be4d74a9c65521d206ef95"
format_stamp: "Formatted at 2025-08-10 13:58:34 on dist-test-slave-k4vt"
I20250810 13:58:34.499902 22286 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250810 13:58:34.506552 22303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:34.508054 22286 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20250810 13:58:34.508474 22286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4709b59128be4d74a9c65521d206ef95"
format_stamp: "Formatted at 2025-08-10 13:58:34 on dist-test-slave-k4vt"
I20250810 13:58:34.508935 22286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:34.570575 22286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:34.572147 22286 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:34.572624 22286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:34.575724 22286 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:34.581086 22286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:34.581303 22286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:34.581581 22286 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:34.581749 22286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:34.748987 22286 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:33745
I20250810 13:58:34.749147 22415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:33745 every 8 connection(s)
I20250810 13:58:34.751550 22286 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:58:34.755568 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 22286
I20250810 13:58:34.756254 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:58:34.765295 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41133
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:34.793007 22416 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:58:34.793584 22416 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:34.794932 22416 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:58:34.798207 22227 ts_manager.cc:194] Registered new tserver with Master: 4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745)
I20250810 13:58:34.801065 22227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:34625
W20250810 13:58:35.082705 22420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:35.083268 22420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:35.083846 22420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:35.116504 22420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:35.117465 22420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:58:35.152992 22420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:35.154402 22420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:35.156112 22420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:35.169183 22426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:35.805466 22416 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
W20250810 13:58:35.173673 22429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:35.170572 22427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:36.373536 22428 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
I20250810 13:58:36.373656 22420 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:36.374883 22420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:36.377800 22420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:36.379277 22420 hybrid_clock.cc:648] HybridClock initialized: now 1754834316379226 us; error 68 us; skew 500 ppm
I20250810 13:58:36.380143 22420 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:36.386811 22420 webserver.cc:489] Webserver started at http://127.19.27.66:33899/ using document root <none> and password file <none>
I20250810 13:58:36.387774 22420 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:36.388044 22420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:36.388542 22420 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:36.393105 22420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e476695322144055a2a13384f8deaff4"
format_stamp: "Formatted at 2025-08-10 13:58:36 on dist-test-slave-k4vt"
I20250810 13:58:36.394306 22420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e476695322144055a2a13384f8deaff4"
format_stamp: "Formatted at 2025-08-10 13:58:36 on dist-test-slave-k4vt"
I20250810 13:58:36.401695 22420 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250810 13:58:36.408102 22436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:36.409475 22420 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 13:58:36.409842 22420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e476695322144055a2a13384f8deaff4"
format_stamp: "Formatted at 2025-08-10 13:58:36 on dist-test-slave-k4vt"
I20250810 13:58:36.410181 22420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:36.479130 22420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:36.480729 22420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:36.481196 22420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:36.483727 22420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:36.487953 22420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:36.488169 22420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:36.488451 22420 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:36.488606 22420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:36.624787 22420 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:44701
I20250810 13:58:36.624895 22548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:44701 every 8 connection(s)
I20250810 13:58:36.627372 22420 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:58:36.632557 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 22420
I20250810 13:58:36.633074 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 13:58:36.639211 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41133
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:36.650784 22549 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:58:36.651414 22549 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:36.652971 22549 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:58:36.655596 22227 ts_manager.cc:194] Registered new tserver with Master: e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
I20250810 13:58:36.656939 22227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:57221
W20250810 13:58:36.958489 22553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:36.959033 22553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:36.959547 22553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:36.992107 22553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:36.992950 22553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:58:37.028977 22553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:37.030330 22553 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:37.032043 22553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:37.045184 22559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:37.661355 22549 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
W20250810 13:58:37.046087 22560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:37.048887 22562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:38.279495 22561 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1229 milliseconds
I20250810 13:58:38.279649 22553 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:38.281049 22553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:38.283231 22553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:38.284577 22553 hybrid_clock.cc:648] HybridClock initialized: now 1754834318284537 us; error 58 us; skew 500 ppm
I20250810 13:58:38.285382 22553 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:38.291527 22553 webserver.cc:489] Webserver started at http://127.19.27.67:34405/ using document root <none> and password file <none>
I20250810 13:58:38.292495 22553 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:38.292716 22553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:38.293248 22553 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:58:38.297808 22553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ee3bf5aa8ab24508a06557a33da2cb5d"
format_stamp: "Formatted at 2025-08-10 13:58:38 on dist-test-slave-k4vt"
I20250810 13:58:38.298960 22553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ee3bf5aa8ab24508a06557a33da2cb5d"
format_stamp: "Formatted at 2025-08-10 13:58:38 on dist-test-slave-k4vt"
I20250810 13:58:38.306636 22553 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250810 13:58:38.312767 22569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:38.313969 22553 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 13:58:38.314445 22553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ee3bf5aa8ab24508a06557a33da2cb5d"
format_stamp: "Formatted at 2025-08-10 13:58:38 on dist-test-slave-k4vt"
I20250810 13:58:38.314832 22553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:38.369065 22553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:38.370563 22553 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:38.371050 22553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:38.373642 22553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:38.377862 22553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:58:38.378095 22553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:38.378371 22553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:58:38.378537 22553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:38.519362 22553 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:37069
I20250810 13:58:38.519464 22681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:37069 every 8 connection(s)
I20250810 13:58:38.522020 22553 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:58:38.527948 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 22553
I20250810 13:58:38.528362 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 13:58:38.557139 22682 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:58:38.557765 22682 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:38.559252 22682 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:58:38.562170 22227 ts_manager.cc:194] Registered new tserver with Master: ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:58:38.563711 22227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:40665
I20250810 13:58:38.566843 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:38.610857 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:38.611218 19565 test_util.cc:276] Using random seed: 875778816
I20250810 13:58:38.654904 22226 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43210:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250810 13:58:38.697883 22484 tablet_service.cc:1468] Processing CreateTablet for tablet 9056c7636f9543a0845e070fb56d24a2 (DEFAULT_TABLE table=TestTable [id=4825261b7d72447b9cda14dc4c444d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:38.699388 22484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9056c7636f9543a0845e070fb56d24a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:38.717978 22702 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap starting.
I20250810 13:58:38.723632 22702 tablet_bootstrap.cc:654] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:38.725445 22702 log.cc:826] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:38.730245 22702 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: No bootstrap required, opened a new log
I20250810 13:58:38.730686 22702 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20250810 13:58:38.749873 22702 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:38.750722 22702 raft_consensus.cc:383] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:38.751029 22702 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e476695322144055a2a13384f8deaff4, State: Initialized, Role: FOLLOWER
I20250810 13:58:38.752015 22702 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:38.752808 22702 raft_consensus.cc:397] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:38.753171 22702 raft_consensus.cc:491] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:38.753607 22702 raft_consensus.cc:3058] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:38.760679 22702 raft_consensus.cc:513] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:38.761659 22702 leader_election.cc:304] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e476695322144055a2a13384f8deaff4; no voters:
I20250810 13:58:38.763829 22702 leader_election.cc:290] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:58:38.764243 22704 raft_consensus.cc:2802] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:38.766898 22704 raft_consensus.cc:695] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 LEADER]: Becoming Leader. State: Replica: e476695322144055a2a13384f8deaff4, State: Running, Role: LEADER
I20250810 13:58:38.767915 22704 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:38.768405 22702 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Time spent starting tablet: real 0.038s user 0.036s sys 0.004s
I20250810 13:58:38.780596 22226 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e476695322144055a2a13384f8deaff4 (127.19.27.66). New cstate: current_term: 1 leader_uuid: "e476695322144055a2a13384f8deaff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } health_report { overall_health: HEALTHY } } }
I20250810 13:58:39.008071 19565 test_util.cc:276] Using random seed: 876175655
I20250810 13:58:39.031316 22223 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43220:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250810 13:58:39.062139 22617 tablet_service.cc:1468] Processing CreateTablet for tablet 0511d933a1a643ec9d645a015855fd42 (DEFAULT_TABLE table=TestTable1 [id=858abbff38294ba6abd1a38a5a387f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:39.063670 22617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0511d933a1a643ec9d645a015855fd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:39.084370 22723 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap starting.
I20250810 13:58:39.090433 22723 tablet_bootstrap.cc:654] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:39.092272 22723 log.cc:826] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:39.096963 22723 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: No bootstrap required, opened a new log
I20250810 13:58:39.097458 22723 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250810 13:58:39.115693 22723 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:39.116357 22723 raft_consensus.cc:383] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:39.116582 22723 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Initialized, Role: FOLLOWER
I20250810 13:58:39.117295 22723 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:39.117821 22723 raft_consensus.cc:397] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:39.118093 22723 raft_consensus.cc:491] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:39.118412 22723 raft_consensus.cc:3058] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:39.122648 22723 raft_consensus.cc:513] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:39.123407 22723 leader_election.cc:304] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3bf5aa8ab24508a06557a33da2cb5d; no voters:
I20250810 13:58:39.125391 22723 leader_election.cc:290] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:58:39.125900 22725 raft_consensus.cc:2802] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:39.128590 22725 raft_consensus.cc:695] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 LEADER]: Becoming Leader. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Running, Role: LEADER
I20250810 13:58:39.128980 22682 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:58:39.129590 22725 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:39.129985 22723 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent starting tablet: real 0.032s user 0.014s sys 0.018s
I20250810 13:58:39.141146 22223 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67). New cstate: current_term: 1 leader_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } health_report { overall_health: HEALTHY } } }
I20250810 13:58:39.351025 19565 test_util.cc:276] Using random seed: 876518610
I20250810 13:58:39.372691 22225 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43224:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250810 13:58:39.403517 22351 tablet_service.cc:1468] Processing CreateTablet for tablet 34b81604d58e46fbbe8cc3cf164b8bdc (DEFAULT_TABLE table=TestTable2 [id=f67742e503c542d78e293dedb21fde69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:58:39.405112 22351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b81604d58e46fbbe8cc3cf164b8bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:39.427856 22744 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:58:39.434314 22744 tablet_bootstrap.cc:654] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Neither blocks nor log segments found. Creating new log.
I20250810 13:58:39.436230 22744 log.cc:826] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:39.441247 22744 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: No bootstrap required, opened a new log
I20250810 13:58:39.441682 22744 ts_tablet_manager.cc:1397] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250810 13:58:39.459899 22744 raft_consensus.cc:357] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:39.460543 22744 raft_consensus.cc:383] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:58:39.460783 22744 raft_consensus.cc:738] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: FOLLOWER
I20250810 13:58:39.461462 22744 consensus_queue.cc:260] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:39.462028 22744 raft_consensus.cc:397] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:39.462311 22744 raft_consensus.cc:491] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:39.462621 22744 raft_consensus.cc:3058] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:58:39.467118 22744 raft_consensus.cc:513] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:39.467981 22744 leader_election.cc:304] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95; no voters:
I20250810 13:58:39.469710 22744 leader_election.cc:290] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:58:39.470113 22746 raft_consensus.cc:2802] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:58:39.472723 22746 raft_consensus.cc:695] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 LEADER]: Becoming Leader. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Running, Role: LEADER
I20250810 13:58:39.473767 22746 consensus_queue.cc:237] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:39.474110 22744 ts_tablet_manager.cc:1428] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.032s user 0.022s sys 0.011s
I20250810 13:58:39.485451 22226 catalog_manager.cc:5582] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4709b59128be4d74a9c65521d206ef95 (127.19.27.65). New cstate: current_term: 1 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } health_report { overall_health: HEALTHY } } }
I20250810 13:58:39.628190 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 22194
W20250810 13:58:39.796510 22549 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41133: connect: Connection refused (error 111)
W20250810 13:58:40.162746 22682 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41133: connect: Connection refused (error 111)
W20250810 13:58:40.506682 22416 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41133: connect: Connection refused (error 111)
I20250810 13:58:44.575312 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 22286
I20250810 13:58:44.599233 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 22420
I20250810 13:58:44.627863 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 22553
I20250810 13:58:44.656024 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41133
--webserver_interface=127.19.27.126
--webserver_port=36591
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:41133 with env {}
W20250810 13:58:44.958279 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:44.958859 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:44.959280 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:44.991488 22823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:58:44.991822 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:44.992096 22823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:58:44.992301 22823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:58:45.028327 22823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:41133
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=36591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:45.029665 22823 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:45.031409 22823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:45.042604 22829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:45.048988 22832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:45.044639 22830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:46.245126 22831 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
I20250810 13:58:46.245249 22823 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:46.246507 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:46.249303 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:46.250759 22823 hybrid_clock.cc:648] HybridClock initialized: now 1754834326250717 us; error 55 us; skew 500 ppm
I20250810 13:58:46.251588 22823 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:46.257869 22823 webserver.cc:489] Webserver started at http://127.19.27.126:36591/ using document root <none> and password file <none>
I20250810 13:58:46.258859 22823 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:46.259112 22823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:46.267382 22823 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250810 13:58:46.272182 22839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:46.273296 22823 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 13:58:46.273695 22823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2d627fde845446419ace1c1815205f13"
format_stamp: "Formatted at 2025-08-10 13:58:32 on dist-test-slave-k4vt"
I20250810 13:58:46.275626 22823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:46.345815 22823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:46.347285 22823 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:46.347764 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:46.418581 22823 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:41133
I20250810 13:58:46.418655 22890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:41133 every 8 connection(s)
I20250810 13:58:46.421499 22823 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:58:46.429471 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 22823
I20250810 13:58:46.431203 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:33745
--local_ip_for_outbound_sockets=127.19.27.65
--tserver_master_addrs=127.19.27.126:41133
--webserver_port=33051
--webserver_interface=127.19.27.65
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:46.431830 22891 sys_catalog.cc:263] Verifying existing consensus state
I20250810 13:58:46.436702 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap starting.
I20250810 13:58:46.451732 22891 log.cc:826] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:46.516887 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:46.517752 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap complete.
I20250810 13:58:46.538828 22891 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:46.541055 22891 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d627fde845446419ace1c1815205f13, State: Initialized, Role: FOLLOWER
I20250810 13:58:46.541824 22891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:46.542492 22891 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:46.542766 22891 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:46.543123 22891 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 2 FOLLOWER]: Advancing to term 3
I20250810 13:58:46.548768 22891 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:46.549463 22891 leader_election.cc:304] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d627fde845446419ace1c1815205f13; no voters:
I20250810 13:58:46.551609 22891 leader_election.cc:290] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250810 13:58:46.552052 22895 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Leader election won for term 3
I20250810 13:58:46.555507 22895 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 LEADER]: Becoming Leader. State: Replica: 2d627fde845446419ace1c1815205f13, State: Running, Role: LEADER
I20250810 13:58:46.556430 22895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:46.556926 22891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:58:46.568174 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d627fde845446419ace1c1815205f13. Latest consensus state: current_term: 3 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:58:46.570036 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:58:46.570896 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:46.574909 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:46.577951 22902 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:58:46.591656 22902 catalog_manager.cc:671] Loaded metadata for table TestTable [id=064f4dc7ba10499e81b2f8788db278d3]
I20250810 13:58:46.593456 22902 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c377de843138483383111d66f8ca28bd]
I20250810 13:58:46.595177 22902 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=f67742e503c542d78e293dedb21fde69]
I20250810 13:58:46.603504 22902 tablet_loader.cc:96] loaded metadata for tablet 0511d933a1a643ec9d645a015855fd42 (table TestTable1 [id=c377de843138483383111d66f8ca28bd])
I20250810 13:58:46.605000 22902 tablet_loader.cc:96] loaded metadata for tablet 34b81604d58e46fbbe8cc3cf164b8bdc (table TestTable2 [id=f67742e503c542d78e293dedb21fde69])
I20250810 13:58:46.606294 22902 tablet_loader.cc:96] loaded metadata for tablet 9056c7636f9543a0845e070fb56d24a2 (table TestTable [id=064f4dc7ba10499e81b2f8788db278d3])
I20250810 13:58:46.607686 22902 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:58:46.612830 22902 catalog_manager.cc:1261] Loaded cluster ID: d97a81b24ea84adeb0c23ddf0223b00c
I20250810 13:58:46.613153 22902 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:58:46.621317 22902 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:58:46.627467 22902 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Loaded TSK: 0
I20250810 13:58:46.629282 22902 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 13:58:46.806083 22893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:46.806619 22893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:46.807135 22893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:46.839298 22893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:46.840308 22893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:58:46.875814 22893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:33745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=33051
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:46.877156 22893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:46.878783 22893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:46.893020 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:48.294978 22917 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 22893
W20250810 13:58:48.736635 22893 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.842s user 0.000s sys 0.004s
W20250810 13:58:48.737267 22893 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.843s user 0.000s sys 0.004s
W20250810 13:58:48.734025 22920 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1840 milliseconds
W20250810 13:58:46.893464 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:48.738857 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:48.738813 22893 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:48.743476 22893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:48.746435 22893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:48.748087 22893 hybrid_clock.cc:648] HybridClock initialized: now 1754834328748032 us; error 62 us; skew 500 ppm
I20250810 13:58:48.749094 22893 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:48.757133 22893 webserver.cc:489] Webserver started at http://127.19.27.65:33051/ using document root <none> and password file <none>
I20250810 13:58:48.758219 22893 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:48.758477 22893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:48.767783 22893 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250810 13:58:48.772991 22928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:48.774142 22893 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 13:58:48.774500 22893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4709b59128be4d74a9c65521d206ef95"
format_stamp: "Formatted at 2025-08-10 13:58:34 on dist-test-slave-k4vt"
I20250810 13:58:48.776553 22893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:48.856074 22893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:48.857585 22893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:48.858080 22893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:48.861311 22893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:48.868010 22935 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 13:58:48.875716 22893 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 13:58:48.876008 22893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250810 13:58:48.876327 22893 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 13:58:48.880997 22893 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 13:58:48.881305 22893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250810 13:58:48.881664 22935 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:58:48.964165 22935 log.cc:826] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:49.071906 22893 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:33745
I20250810 13:58:49.072172 23042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:33745 every 8 connection(s)
I20250810 13:58:49.075644 22893 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:58:49.083943 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 22893
I20250810 13:58:49.085762 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:44701
--local_ip_for_outbound_sockets=127.19.27.66
--tserver_master_addrs=127.19.27.126:41133
--webserver_port=33899
--webserver_interface=127.19.27.66
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:49.099670 22935 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:49.100852 22935 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap complete.
I20250810 13:58:49.102608 22935 ts_tablet_manager.cc:1397] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.221s user 0.151s sys 0.063s
I20250810 13:58:49.123140 23043 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:58:49.123677 23043 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:49.121466 22935 raft_consensus.cc:357] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:49.124478 22935 raft_consensus.cc:738] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: FOLLOWER
I20250810 13:58:49.124959 23043 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:58:49.125468 22935 consensus_queue.cc:260] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:49.127492 22935 raft_consensus.cc:397] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:49.127925 22935 raft_consensus.cc:491] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:49.128335 22935 raft_consensus.cc:3058] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:58:49.130167 22856 ts_manager.cc:194] Registered new tserver with Master: 4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745)
I20250810 13:58:49.137085 22935 raft_consensus.cc:513] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:49.138008 22856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:49081
I20250810 13:58:49.138038 22935 leader_election.cc:304] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95; no voters:
I20250810 13:58:49.144446 23048 raft_consensus.cc:2802] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 13:58:49.145704 22935 leader_election.cc:290] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 13:58:49.152974 23043 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:58:49.152932 23048 raft_consensus.cc:695] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 LEADER]: Becoming Leader. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Running, Role: LEADER
I20250810 13:58:49.153926 23048 consensus_queue.cc:237] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:58:49.160313 22935 ts_tablet_manager.cc:1428] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.057s user 0.041s sys 0.008s
I20250810 13:58:49.172505 22856 catalog_manager.cc:5582] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } health_report { overall_health: HEALTHY } } }
W20250810 13:58:49.461581 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:49.462119 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:49.462627 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:49.496039 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:49.496892 23047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:58:49.532281 23047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:44701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=33899
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:49.533643 23047 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:49.535344 23047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:49.548298 23062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:49.549811 23063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:50.767675 23047 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.220s user 0.458s sys 0.761s
W20250810 13:58:50.768106 23047 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.221s user 0.460s sys 0.761s
I20250810 13:58:50.769618 23047 server_base.cc:1047] running on GCE node
W20250810 13:58:50.771206 23067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:50.772446 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:50.774746 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:50.776126 23047 hybrid_clock.cc:648] HybridClock initialized: now 1754834330776088 us; error 53 us; skew 500 ppm
I20250810 13:58:50.776960 23047 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:50.784168 23047 webserver.cc:489] Webserver started at http://127.19.27.66:33899/ using document root <none> and password file <none>
I20250810 13:58:50.785247 23047 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:50.785634 23047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:50.794137 23047 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20250810 13:58:50.799585 23072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:50.800908 23047 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250810 13:58:50.801256 23047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e476695322144055a2a13384f8deaff4"
format_stamp: "Formatted at 2025-08-10 13:58:36 on dist-test-slave-k4vt"
I20250810 13:58:50.803239 23047 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:50.872210 23047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:50.873680 23047 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:50.874133 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:50.876643 23047 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:50.882174 23079 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 13:58:50.890028 23047 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 13:58:50.890304 23047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250810 13:58:50.890590 23047 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 13:58:50.895295 23047 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 13:58:50.895493 23047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250810 13:58:50.895923 23079 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap starting.
I20250810 13:58:50.951314 23079 log.cc:826] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:51.058435 23047 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:44701
I20250810 13:58:51.058562 23186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:44701 every 8 connection(s)
I20250810 13:58:51.062222 23047 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:58:51.066867 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 23047
I20250810 13:58:51.068792 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:37069
--local_ip_for_outbound_sockets=127.19.27.67
--tserver_master_addrs=127.19.27.126:41133
--webserver_port=34405
--webserver_interface=127.19.27.67
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:58:51.100749 23079 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:51.101864 23079 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap complete.
I20250810 13:58:51.102938 23187 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:58:51.103376 23079 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Time spent bootstrapping tablet: real 0.208s user 0.153s sys 0.051s
I20250810 13:58:51.103453 23187 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:51.104844 23187 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:58:51.109021 22856 ts_manager.cc:194] Registered new tserver with Master: e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
I20250810 13:58:51.112329 22856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:41017
I20250810 13:58:51.117870 23079 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:51.119817 23079 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e476695322144055a2a13384f8deaff4, State: Initialized, Role: FOLLOWER
I20250810 13:58:51.120568 23079 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:51.121043 23079 raft_consensus.cc:397] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:51.121280 23079 raft_consensus.cc:491] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:51.121555 23079 raft_consensus.cc:3058] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:58:51.126520 23079 raft_consensus.cc:513] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:51.127161 23079 leader_election.cc:304] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e476695322144055a2a13384f8deaff4; no voters:
I20250810 13:58:51.128836 23079 leader_election.cc:290] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 13:58:51.129309 23192 raft_consensus.cc:2802] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 13:58:51.132084 23187 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:58:51.132782 23079 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Time spent starting tablet: real 0.029s user 0.026s sys 0.005s
I20250810 13:58:51.133397 23192 raft_consensus.cc:695] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 LEADER]: Becoming Leader. State: Replica: e476695322144055a2a13384f8deaff4, State: Running, Role: LEADER
I20250810 13:58:51.134261 23192 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } }
I20250810 13:58:51.144212 22856 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 reported cstate change: term changed from 0 to 2, leader changed from <none> to e476695322144055a2a13384f8deaff4 (127.19.27.66), VOTER e476695322144055a2a13384f8deaff4 (127.19.27.66) added. New cstate: current_term: 2 leader_uuid: "e476695322144055a2a13384f8deaff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } health_report { overall_health: HEALTHY } } }
I20250810 13:58:51.172545 23142 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:51.175884 23192 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } } }
I20250810 13:58:51.185159 22841 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9056c7636f9543a0845e070fb56d24a2 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250810 13:58:51.188892 22856 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 reported cstate change: config changed from index -1 to 10, NON_VOTER 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) added. New cstate: current_term: 2 leader_uuid: "e476695322144055a2a13384f8deaff4" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250810 13:58:51.191059 23075 consensus_peers.cc:489] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 -> Peer 4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745): Couldn't send request to peer 4709b59128be4d74a9c65521d206ef95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9056c7636f9543a0845e070fb56d24a2. This is attempt 1: this message will repeat every 5th retry.
W20250810 13:58:51.196707 22856 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9056c7636f9543a0845e070fb56d24a2 with cas_config_opid_index 10: no extra replica candidate found for tablet 9056c7636f9543a0845e070fb56d24a2 (table TestTable [id=064f4dc7ba10499e81b2f8788db278d3]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250810 13:58:51.411288 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:58:51.411839 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:58:51.412329 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:58:51.443645 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:58:51.444519 23191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:58:51.481026 23191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:37069
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=34405
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:58:51.482419 23191 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:51.484218 23191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:51.497160 23209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:51.713153 23216 ts_tablet_manager.cc:927] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Initiating tablet copy from peer e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
I20250810 13:58:51.722879 23216 tablet_copy_client.cc:323] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Beginning tablet copy session from remote peer at address 127.19.27.66:44701
I20250810 13:58:51.774296 23162 tablet_copy_service.cc:140] P e476695322144055a2a13384f8deaff4: Received BeginTabletCopySession request for tablet 9056c7636f9543a0845e070fb56d24a2 from peer 4709b59128be4d74a9c65521d206ef95 ({username='slave'} at 127.19.27.65:51197)
I20250810 13:58:51.775108 23162 tablet_copy_service.cc:161] P e476695322144055a2a13384f8deaff4: Beginning new tablet copy session on tablet 9056c7636f9543a0845e070fb56d24a2 from peer 4709b59128be4d74a9c65521d206ef95 at {username='slave'} at 127.19.27.65:51197: session id = 4709b59128be4d74a9c65521d206ef95-9056c7636f9543a0845e070fb56d24a2
I20250810 13:58:51.785741 23162 tablet_copy_source_session.cc:215] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 13:58:51.791904 23216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9056c7636f9543a0845e070fb56d24a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:51.815838 23216 tablet_copy_client.cc:806] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Starting download of 0 data blocks...
I20250810 13:58:51.816790 23216 tablet_copy_client.cc:670] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Starting download of 1 WAL segments...
I20250810 13:58:51.823297 23216 tablet_copy_client.cc:538] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 13:58:51.835552 23216 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:58:52.037216 23216 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:52.038542 23216 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Bootstrap complete.
I20250810 13:58:52.039419 23216 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.204s user 0.181s sys 0.025s
I20250810 13:58:52.042685 23216 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:52.043617 23216 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: LEARNER
I20250810 13:58:52.044435 23216 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:52.054625 23216 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20250810 13:58:52.071884 23162 tablet_copy_service.cc:342] P e476695322144055a2a13384f8deaff4: Request end of tablet copy session 4709b59128be4d74a9c65521d206ef95-9056c7636f9543a0845e070fb56d24a2 received from {username='slave'} at 127.19.27.65:51197
I20250810 13:58:52.072340 23162 tablet_copy_service.cc:434] P e476695322144055a2a13384f8deaff4: ending tablet copy session 4709b59128be4d74a9c65521d206ef95-9056c7636f9543a0845e070fb56d24a2 on tablet 9056c7636f9543a0845e070fb56d24a2 with peer 4709b59128be4d74a9c65521d206ef95
I20250810 13:58:52.345645 22998 raft_consensus.cc:1215] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250810 13:58:52.767428 23222 raft_consensus.cc:1062] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: attempting to promote NON_VOTER 4709b59128be4d74a9c65521d206ef95 to VOTER
I20250810 13:58:52.770386 23222 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:58:52.786684 22998 raft_consensus.cc:1273] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Refusing update from remote peer e476695322144055a2a13384f8deaff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250810 13:58:52.789330 23223 consensus_queue.cc:1035] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250810 13:58:52.808171 23223 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:52.823623 22998 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:52.840857 22856 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 reported cstate change: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e476695322144055a2a13384f8deaff4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250810 13:58:52.900985 23208 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23191
W20250810 13:58:53.189764 23191 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.692s user 0.603s sys 1.081s
W20250810 13:58:53.190220 23191 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.693s user 0.603s sys 1.081s
W20250810 13:58:51.497880 23210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:53.191848 23212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:53.194573 23211 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1693 milliseconds
I20250810 13:58:53.194641 23191 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:58:53.195760 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:58:53.197791 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:58:53.199141 23191 hybrid_clock.cc:648] HybridClock initialized: now 1754834333199103 us; error 47 us; skew 500 ppm
I20250810 13:58:53.199954 23191 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:53.205982 23191 webserver.cc:489] Webserver started at http://127.19.27.67:34405/ using document root <none> and password file <none>
I20250810 13:58:53.206912 23191 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:53.207154 23191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:53.214818 23191 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250810 13:58:53.219560 23235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:53.220744 23191 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250810 13:58:53.221092 23191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ee3bf5aa8ab24508a06557a33da2cb5d"
format_stamp: "Formatted at 2025-08-10 13:58:38 on dist-test-slave-k4vt"
I20250810 13:58:53.222977 23191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:53.279850 23191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:53.281376 23191 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:53.281824 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:53.284456 23191 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:58:53.290300 23242 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 13:58:53.298189 23191 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 13:58:53.298467 23191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250810 13:58:53.298774 23191 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 13:58:53.303534 23191 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 13:58:53.303740 23191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250810 13:58:53.304198 23242 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap starting.
I20250810 13:58:53.360581 23242 log.cc:826] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:53.470696 23242 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:53.471967 23242 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap complete.
I20250810 13:58:53.473834 23242 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent bootstrapping tablet: real 0.170s user 0.114s sys 0.050s
I20250810 13:58:53.491619 23242 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:53.493870 23242 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Initialized, Role: FOLLOWER
I20250810 13:58:53.494935 23242 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:53.495736 23242 raft_consensus.cc:397] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:53.496126 23242 raft_consensus.cc:491] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:53.496555 23242 raft_consensus.cc:3058] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 1 FOLLOWER]: Advancing to term 2
I20250810 13:58:53.497798 22841 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9056c7636f9543a0845e070fb56d24a2 with cas_config_opid_index 10: aborting the task: latest config opid_index 11; task opid_index 10
I20250810 13:58:53.502740 23242 raft_consensus.cc:513] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:53.503701 23242 leader_election.cc:304] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3bf5aa8ab24508a06557a33da2cb5d; no voters:
I20250810 13:58:53.505978 23242 leader_election.cc:290] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 13:58:53.506372 23349 raft_consensus.cc:2802] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Leader election won for term 2
I20250810 13:58:53.510823 23191 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:37069
I20250810 13:58:53.511348 23353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:37069 every 8 connection(s)
I20250810 13:58:53.511463 23242 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent starting tablet: real 0.037s user 0.029s sys 0.005s
I20250810 13:58:53.513020 23349 raft_consensus.cc:695] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEADER]: Becoming Leader. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Running, Role: LEADER
I20250810 13:58:53.513757 23191 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:58:53.514504 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 23191
I20250810 13:58:53.515702 23349 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } }
I20250810 13:58:53.544535 23354 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:58:53.544991 23354 heartbeater.cc:461] Registering TS with master...
I20250810 13:58:53.546028 23354 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:58:53.549463 22855 ts_manager.cc:194] Registered new tserver with Master: ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:58:53.550427 22855 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d reported cstate change: term changed from 0 to 2, leader changed from <none> to ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67), VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) added. New cstate: current_term: 2 leader_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } health_report { overall_health: HEALTHY } } }
I20250810 13:58:53.552081 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:58:53.557210 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250810 13:58:53.561117 19565 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250810 13:58:53.562899 22855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:60079
I20250810 13:58:53.566829 23354 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:58:53.576200 23306 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } }
I20250810 13:58:53.579330 23352 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER e476695322144055a2a13384f8deaff4 (127.19.27.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } }
I20250810 13:58:53.587720 22840 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0511d933a1a643ec9d645a015855fd42 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250810 13:58:53.589787 23237 consensus_peers.cc:489] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d -> Peer e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): Couldn't send request to peer e476695322144055a2a13384f8deaff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0511d933a1a643ec9d645a015855fd42. This is attempt 1: this message will repeat every 5th retry.
I20250810 13:58:53.590761 22855 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d reported cstate change: config changed from index -1 to 9, NON_VOTER e476695322144055a2a13384f8deaff4 (127.19.27.66) added. New cstate: current_term: 2 leader_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250810 13:58:53.600018 23306 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:53.603672 23349 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } } }
W20250810 13:58:53.605361 23237 consensus_peers.cc:489] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d -> Peer e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): Couldn't send request to peer e476695322144055a2a13384f8deaff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0511d933a1a643ec9d645a015855fd42. This is attempt 1: this message will repeat every 5th retry.
I20250810 13:58:53.612061 22840 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0511d933a1a643ec9d645a015855fd42 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250810 13:58:53.615829 22855 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d reported cstate change: config changed from index 9 to 10, NON_VOTER 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) added. New cstate: current_term: 2 leader_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250810 13:58:53.616606 23238 consensus_peers.cc:489] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d -> Peer 4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745): Couldn't send request to peer 4709b59128be4d74a9c65521d206ef95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0511d933a1a643ec9d645a015855fd42. This is attempt 1: this message will repeat every 5th retry.
I20250810 13:58:54.033092 23142 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: NON_VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: true } }
I20250810 13:58:54.040445 22998 raft_consensus.cc:1273] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Refusing update from remote peer e476695322144055a2a13384f8deaff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250810 13:58:54.041685 23371 consensus_queue.cc:1035] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250810 13:58:54.048655 23369 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: NON_VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: true } } }
W20250810 13:58:54.050392 23073 consensus_peers.cc:489] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 -> Peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Couldn't send request to peer ee3bf5aa8ab24508a06557a33da2cb5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9056c7636f9543a0845e070fb56d24a2. This is attempt 1: this message will repeat every 5th retry.
I20250810 13:58:54.050215 22998 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: NON_VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: true } } }
I20250810 13:58:54.059163 22841 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9056c7636f9543a0845e070fb56d24a2 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 7)
I20250810 13:58:54.061841 22856 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: config changed from index 11 to 12, NON_VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) added. New cstate: current_term: 2 leader_uuid: "e476695322144055a2a13384f8deaff4" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: NON_VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: true } } }
I20250810 13:58:54.082286 23376 ts_tablet_manager.cc:927] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Initiating tablet copy from peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:58:54.083957 23376 tablet_copy_client.cc:323] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Beginning tablet copy session from remote peer at address 127.19.27.67:37069
I20250810 13:58:54.096125 23326 tablet_copy_service.cc:140] P ee3bf5aa8ab24508a06557a33da2cb5d: Received BeginTabletCopySession request for tablet 0511d933a1a643ec9d645a015855fd42 from peer 4709b59128be4d74a9c65521d206ef95 ({username='slave'} at 127.19.27.65:39073)
I20250810 13:58:54.096700 23326 tablet_copy_service.cc:161] P ee3bf5aa8ab24508a06557a33da2cb5d: Beginning new tablet copy session on tablet 0511d933a1a643ec9d645a015855fd42 from peer 4709b59128be4d74a9c65521d206ef95 at {username='slave'} at 127.19.27.65:39073: session id = 4709b59128be4d74a9c65521d206ef95-0511d933a1a643ec9d645a015855fd42
I20250810 13:58:54.101327 23326 tablet_copy_source_session.cc:215] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 13:58:54.104647 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0511d933a1a643ec9d645a015855fd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:54.115931 23376 tablet_copy_client.cc:806] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Starting download of 0 data blocks...
I20250810 13:58:54.116410 23376 tablet_copy_client.cc:670] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Starting download of 1 WAL segments...
I20250810 13:58:54.119841 23376 tablet_copy_client.cc:538] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 13:58:54.125041 23376 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:58:54.203580 23380 ts_tablet_manager.cc:927] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Initiating tablet copy from peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:58:54.205776 23380 tablet_copy_client.cc:323] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: tablet copy: Beginning tablet copy session from remote peer at address 127.19.27.67:37069
I20250810 13:58:54.206357 23376 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:54.206940 23376 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Bootstrap complete.
I20250810 13:58:54.207262 23326 tablet_copy_service.cc:140] P ee3bf5aa8ab24508a06557a33da2cb5d: Received BeginTabletCopySession request for tablet 0511d933a1a643ec9d645a015855fd42 from peer e476695322144055a2a13384f8deaff4 ({username='slave'} at 127.19.27.66:42793)
I20250810 13:58:54.207366 23376 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.082s user 0.070s sys 0.010s
I20250810 13:58:54.207649 23326 tablet_copy_service.cc:161] P ee3bf5aa8ab24508a06557a33da2cb5d: Beginning new tablet copy session on tablet 0511d933a1a643ec9d645a015855fd42 from peer e476695322144055a2a13384f8deaff4 at {username='slave'} at 127.19.27.66:42793: session id = e476695322144055a2a13384f8deaff4-0511d933a1a643ec9d645a015855fd42
I20250810 13:58:54.209012 23376 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:54.209542 23376 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: LEARNER
I20250810 13:58:54.209931 23376 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:54.211572 23376 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250810 13:58:54.212540 23326 tablet_copy_source_session.cc:215] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 13:58:54.213138 23325 tablet_copy_service.cc:342] P ee3bf5aa8ab24508a06557a33da2cb5d: Request end of tablet copy session 4709b59128be4d74a9c65521d206ef95-0511d933a1a643ec9d645a015855fd42 received from {username='slave'} at 127.19.27.65:39073
I20250810 13:58:54.213572 23325 tablet_copy_service.cc:434] P ee3bf5aa8ab24508a06557a33da2cb5d: ending tablet copy session 4709b59128be4d74a9c65521d206ef95-0511d933a1a643ec9d645a015855fd42 on tablet 0511d933a1a643ec9d645a015855fd42 with peer 4709b59128be4d74a9c65521d206ef95
I20250810 13:58:54.215580 23380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0511d933a1a643ec9d645a015855fd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:54.225991 23380 tablet_copy_client.cc:806] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: tablet copy: Starting download of 0 data blocks...
I20250810 13:58:54.226502 23380 tablet_copy_client.cc:670] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: tablet copy: Starting download of 1 WAL segments...
I20250810 13:58:54.229940 23380 tablet_copy_client.cc:538] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 13:58:54.235450 23380 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Bootstrap starting.
I20250810 13:58:54.317569 23380 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:54.318220 23380 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Bootstrap complete.
I20250810 13:58:54.318668 23380 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Time spent bootstrapping tablet: real 0.083s user 0.064s sys 0.020s
I20250810 13:58:54.320391 23380 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:54.320870 23380 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e476695322144055a2a13384f8deaff4, State: Initialized, Role: LEARNER
I20250810 13:58:54.321308 23380 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: NON_VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: true } }
I20250810 13:58:54.322577 23380 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250810 13:58:54.324151 23325 tablet_copy_service.cc:342] P ee3bf5aa8ab24508a06557a33da2cb5d: Request end of tablet copy session e476695322144055a2a13384f8deaff4-0511d933a1a643ec9d645a015855fd42 received from {username='slave'} at 127.19.27.66:42793
I20250810 13:58:54.324479 23325 tablet_copy_service.cc:434] P ee3bf5aa8ab24508a06557a33da2cb5d: ending tablet copy session e476695322144055a2a13384f8deaff4-0511d933a1a643ec9d645a015855fd42 on tablet 0511d933a1a643ec9d645a015855fd42 with peer e476695322144055a2a13384f8deaff4
I20250810 13:58:54.531101 22998 raft_consensus.cc:1215] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250810 13:58:54.562958 23386 ts_tablet_manager.cc:927] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Initiating tablet copy from peer e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
I20250810 13:58:54.565097 23386 tablet_copy_client.cc:323] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.27.66:44701
I20250810 13:58:54.566705 23162 tablet_copy_service.cc:140] P e476695322144055a2a13384f8deaff4: Received BeginTabletCopySession request for tablet 9056c7636f9543a0845e070fb56d24a2 from peer ee3bf5aa8ab24508a06557a33da2cb5d ({username='slave'} at 127.19.27.67:43117)
I20250810 13:58:54.566687 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4709b59128be4d74a9c65521d206ef95 to finish bootstrapping
I20250810 13:58:54.567198 23162 tablet_copy_service.cc:161] P e476695322144055a2a13384f8deaff4: Beginning new tablet copy session on tablet 9056c7636f9543a0845e070fb56d24a2 from peer ee3bf5aa8ab24508a06557a33da2cb5d at {username='slave'} at 127.19.27.67:43117: session id = ee3bf5aa8ab24508a06557a33da2cb5d-9056c7636f9543a0845e070fb56d24a2
I20250810 13:58:54.573499 23162 tablet_copy_source_session.cc:215] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 13:58:54.576514 23386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9056c7636f9543a0845e070fb56d24a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:58:54.587194 23386 tablet_copy_client.cc:806] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: tablet copy: Starting download of 0 data blocks...
I20250810 13:58:54.587745 23386 tablet_copy_client.cc:670] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: tablet copy: Starting download of 1 WAL segments...
I20250810 13:58:54.589241 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e476695322144055a2a13384f8deaff4 to finish bootstrapping
I20250810 13:58:54.592399 23386 tablet_copy_client.cc:538] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 13:58:54.599346 23386 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap starting.
I20250810 13:58:54.601989 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ee3bf5aa8ab24508a06557a33da2cb5d to finish bootstrapping
I20250810 13:58:54.698639 23386 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:54.699375 23386 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap complete.
I20250810 13:58:54.699959 23386 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent bootstrapping tablet: real 0.101s user 0.091s sys 0.012s
I20250810 13:58:54.701058 23142 raft_consensus.cc:1215] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250810 13:58:54.701892 23386 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: NON_VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: true } }
I20250810 13:58:54.702638 23386 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Initialized, Role: LEARNER
I20250810 13:58:54.703197 23386 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: NON_VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: true } }
I20250810 13:58:54.704952 23386 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250810 13:58:54.706743 23162 tablet_copy_service.cc:342] P e476695322144055a2a13384f8deaff4: Request end of tablet copy session ee3bf5aa8ab24508a06557a33da2cb5d-9056c7636f9543a0845e070fb56d24a2 received from {username='slave'} at 127.19.27.67:43117
I20250810 13:58:54.707101 23162 tablet_copy_service.cc:434] P e476695322144055a2a13384f8deaff4: ending tablet copy session ee3bf5aa8ab24508a06557a33da2cb5d-9056c7636f9543a0845e070fb56d24a2 on tablet 9056c7636f9543a0845e070fb56d24a2 with peer ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:58:54.943599 23384 raft_consensus.cc:1062] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: attempting to promote NON_VOTER 4709b59128be4d74a9c65521d206ef95 to VOTER
I20250810 13:58:54.945932 23384 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:58:54.954177 22998 raft_consensus.cc:1273] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 LEARNER]: Refusing update from remote peer ee3bf5aa8ab24508a06557a33da2cb5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250810 13:58:54.955837 23385 consensus_queue.cc:1035] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250810 13:58:54.957557 23142 raft_consensus.cc:1273] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 LEARNER]: Refusing update from remote peer ee3bf5aa8ab24508a06557a33da2cb5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250810 13:58:54.959079 23385 consensus_queue.cc:1035] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250810 13:58:54.966490 23285 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:58:54.967005 22977 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:58:54.974236 23122 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:58:54.975574 23385 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:54.979389 23142 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:54.981276 22997 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:55.000780 22854 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 reported cstate change: config changed from index 10 to 11, 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: NON_VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:55.011070 23349 raft_consensus.cc:1062] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: attempting to promote NON_VOTER e476695322144055a2a13384f8deaff4 to VOTER
I20250810 13:58:55.013120 23349 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:58:55.020185 22998 raft_consensus.cc:1273] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Refusing update from remote peer ee3bf5aa8ab24508a06557a33da2cb5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250810 13:58:55.021375 23142 raft_consensus.cc:1273] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 LEARNER]: Refusing update from remote peer ee3bf5aa8ab24508a06557a33da2cb5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250810 13:58:55.023546 23384 consensus_queue.cc:1035] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250810 13:58:55.025935 23349 consensus_queue.cc:1035] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250810 13:58:55.036391 23384 raft_consensus.cc:1025] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEADER]: attempt to promote peer e476695322144055a2a13384f8deaff4: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250810 13:58:55.039520 22997 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e476695322144055a2a13384f8deaff4 (127.19.27.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:55.040633 23142 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e476695322144055a2a13384f8deaff4 (127.19.27.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:55.044020 23385 raft_consensus.cc:2953] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e476695322144055a2a13384f8deaff4 (127.19.27.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:55.073680 22855 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: config changed from index 11 to 12, e476695322144055a2a13384f8deaff4 (127.19.27.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:58:55.185498 23306 raft_consensus.cc:1215] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
2d627fde845446419ace1c1815205f13 | 127.19.27.126:41133 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:40003 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4709b59128be4d74a9c65521d206ef95 | 127.19.27.65:33745 | HEALTHY | <none> | 1 | 0
e476695322144055a2a13384f8deaff4 | 127.19.27.66:44701 | HEALTHY | <none> | 1 | 0
ee3bf5aa8ab24508a06557a33da2cb5d | 127.19.27.67:37069 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.19.27.65 | experimental | 127.19.27.65:33745
local_ip_for_outbound_sockets | 127.19.27.66 | experimental | 127.19.27.66:44701
local_ip_for_outbound_sockets | 127.19.27.67 | experimental | 127.19.27.67:37069
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.19.27.65:33745
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.27.66:44701
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.27.67:37069
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:40003 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 9056c7636f9543a0845e070fb56d24a2 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): RUNNING [LEADER]
4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745): RUNNING
ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): RUNNING [NONVOTER]
All reported replicas are:
A = e476695322144055a2a13384f8deaff4
B = 4709b59128be4d74a9c65521d206ef95
C = ee3bf5aa8ab24508a06557a33da2cb5d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 12 | Yes
B | A* B C~ | 2 | 12 | Yes
C | A B C~ | 2 | 12 | Yes
Tablet 0511d933a1a643ec9d645a015855fd42 of table 'TestTable1' is conflicted: 3 replicas' active configs disagree with the leader master's
ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): RUNNING [LEADER]
e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): RUNNING [NONVOTER]
4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745): RUNNING [NONVOTER]
All reported replicas are:
A = ee3bf5aa8ab24508a06557a33da2cb5d
B = e476695322144055a2a13384f8deaff4
C = 4709b59128be4d74a9c65521d206ef95
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B~ C~ | | | Yes
A | A* B~ C | 2 | 11 | No
B | A* B~ C | 2 | 11 | Yes
C | A* B~ C | 2 | 11 | No
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
TestTable1 | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 2 out of 3 table(s) are not healthy
FAILED
I20250810 13:58:55.294152 19565 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250810 13:58:55.620687 23427 raft_consensus.cc:1062] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: attempting to promote NON_VOTER ee3bf5aa8ab24508a06557a33da2cb5d to VOTER
I20250810 13:58:55.622222 23427 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:58:55.627550 23306 raft_consensus.cc:1273] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 LEARNER]: Refusing update from remote peer e476695322144055a2a13384f8deaff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250810 13:58:55.627534 22997 raft_consensus.cc:1273] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Refusing update from remote peer e476695322144055a2a13384f8deaff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250810 13:58:55.629074 23427 consensus_queue.cc:1035] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250810 13:58:55.630445 23374 consensus_queue.cc:1035] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250810 13:58:55.639132 23426 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } } }
I20250810 13:58:55.640560 22997 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } } }
I20250810 13:58:55.641142 23306 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } } }
I20250810 13:58:55.653901 22855 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d reported cstate change: config changed from index 12 to 13, ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e476695322144055a2a13384f8deaff4" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } } }
I20250810 13:58:55.759490 23122 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:58:55.762473 22977 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:58:55.770076 23285 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
2d627fde845446419ace1c1815205f13 | 127.19.27.126:41133 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:40003 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4709b59128be4d74a9c65521d206ef95 | 127.19.27.65:33745 | HEALTHY | <none> | 1 | 0
e476695322144055a2a13384f8deaff4 | 127.19.27.66:44701 | HEALTHY | <none> | 1 | 0
ee3bf5aa8ab24508a06557a33da2cb5d | 127.19.27.67:37069 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.19.27.65 | experimental | 127.19.27.65:33745
local_ip_for_outbound_sockets | 127.19.27.66 | experimental | 127.19.27.66:44701
local_ip_for_outbound_sockets | 127.19.27.67 | experimental | 127.19.27.67:37069
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.19.27.65:33745
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.27.66:44701
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.27.67:37069
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:40003 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250810 13:58:55.987768 19565 log_verifier.cc:126] Checking tablet 0511d933a1a643ec9d645a015855fd42
I20250810 13:58:56.082736 19565 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 0511d933a1a643ec9d645a015855fd42
I20250810 13:58:56.083070 19565 log_verifier.cc:126] Checking tablet 34b81604d58e46fbbe8cc3cf164b8bdc
I20250810 13:58:56.109681 19565 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 34b81604d58e46fbbe8cc3cf164b8bdc
I20250810 13:58:56.109907 19565 log_verifier.cc:126] Checking tablet 9056c7636f9543a0845e070fb56d24a2
I20250810 13:58:56.200455 19565 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 9056c7636f9543a0845e070fb56d24a2
I20250810 13:58:56.200920 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 22823
I20250810 13:58:56.227905 19565 minidump.cc:252] Setting minidump size limit to 20M
I20250810 13:58:56.229254 19565 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:58:56.230676 19565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:58:56.243185 23464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:56.243216 23465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:58:56.346580 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:58:56.346840 19565 server_base.cc:1047] running on GCE node
I20250810 13:58:56.348099 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250810 13:58:56.348345 19565 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250810 13:58:56.348521 19565 hybrid_clock.cc:648] HybridClock initialized: now 1754834336348498 us; error 0 us; skew 500 ppm
I20250810 13:58:56.349167 19565 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:58:56.352185 19565 webserver.cc:489] Webserver started at http://0.0.0.0:45557/ using document root <none> and password file <none>
I20250810 13:58:56.353032 19565 fs_manager.cc:362] Metadata directory not provided
I20250810 13:58:56.353288 19565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:58:56.359030 19565 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20250810 13:58:56.362823 23472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:58:56.363824 19565 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250810 13:58:56.364151 19565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2d627fde845446419ace1c1815205f13"
format_stamp: "Formatted at 2025-08-10 13:58:32 on dist-test-slave-k4vt"
I20250810 13:58:56.365947 19565 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:58:56.386305 19565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:58:56.387748 19565 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:58:56.388279 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:58:56.397049 19565 sys_catalog.cc:263] Verifying existing consensus state
W20250810 13:58:56.401098 19565 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.19.27.126:41133 exists but no master address supplied!
I20250810 13:58:56.402994 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap starting.
I20250810 13:58:56.444177 19565 log.cc:826] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Log is configured to *not* fsync() on all Append() calls
I20250810 13:58:56.550103 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:58:56.550933 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap complete.
I20250810 13:58:56.564711 19565 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:56.565465 19565 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d627fde845446419ace1c1815205f13, State: Initialized, Role: FOLLOWER
I20250810 13:58:56.566402 19565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:56.567078 19565 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:58:56.567420 19565 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:58:56.567840 19565 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 3 FOLLOWER]: Advancing to term 4
I20250810 13:58:56.574085 19565 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:56.574877 19565 leader_election.cc:304] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d627fde845446419ace1c1815205f13; no voters:
I20250810 13:58:56.576072 19565 leader_election.cc:290] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250810 13:58:56.576336 23479 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 4 FOLLOWER]: Leader election won for term 4
I20250810 13:58:56.577726 23479 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 4 LEADER]: Becoming Leader. State: Replica: 2d627fde845446419ace1c1815205f13, State: Running, Role: LEADER
I20250810 13:58:56.578590 23479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:58:56.586004 23481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d627fde845446419ace1c1815205f13. Latest consensus state: current_term: 4 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:58:56.586730 23481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:56.586788 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:58:56.587316 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:58:56.614718 19565 tablet_replica.cc:331] stopping tablet replica
I20250810 13:58:56.615293 19565 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 4 LEADER]: Raft consensus shutting down.
I20250810 13:58:56.615707 19565 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250810 13:58:56.617935 19565 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250810 13:58:56.618400 19565 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250810 13:58:56.651202 19565 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250810 13:58:56.676373 23043 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41133: connect: Connection refused (error 111)
W20250810 13:58:56.678001 23354 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41133: connect: Connection refused (error 111)
W20250810 13:58:56.679104 23187 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:41133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:41133: connect: Connection refused (error 111)
W20250810 13:58:57.523912 23039 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5400 after lost signal to thread 22913
W20250810 13:58:57.525327 23039 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5c40 after lost signal to thread 23042
I20250810 13:59:02.292032 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 22893
I20250810 13:59:02.321333 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 23047
W20250810 13:59:02.321312 23238 connection.cc:537] client connection to 127.19.27.65:33745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250810 13:59:02.321893 23238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250810 13:59:02.332607 23238 consensus_peers.cc:489] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d -> Peer 4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745): Couldn't send request to peer 4709b59128be4d74a9c65521d206ef95. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.65:33745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 13:59:02.353458 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 23191
I20250810 13:59:02.385564 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41133
--webserver_interface=127.19.27.126
--webserver_port=36591
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:41133 with env {}
W20250810 13:59:02.692255 23560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:02.692904 23560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:02.693377 23560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:02.724260 23560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:59:02.724766 23560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:02.725155 23560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:59:02.725507 23560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:59:02.761878 23560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:41133
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=36591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:02.763242 23560 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:02.764948 23560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:02.776695 23566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:02.776965 23567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:03.950206 23569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:03.952204 23568 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1172 milliseconds
I20250810 13:59:03.952322 23560 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:03.953613 23560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:03.956246 23560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:03.957595 23560 hybrid_clock.cc:648] HybridClock initialized: now 1754834343957565 us; error 48 us; skew 500 ppm
I20250810 13:59:03.958415 23560 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:03.964726 23560 webserver.cc:489] Webserver started at http://127.19.27.126:36591/ using document root <none> and password file <none>
I20250810 13:59:03.965696 23560 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:03.965955 23560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:03.974066 23560 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250810 13:59:03.978780 23576 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:03.979926 23560 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250810 13:59:03.980293 23560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2d627fde845446419ace1c1815205f13"
format_stamp: "Formatted at 2025-08-10 13:58:32 on dist-test-slave-k4vt"
I20250810 13:59:03.982426 23560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:04.038311 23560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:04.040021 23560 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:04.040499 23560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:04.112934 23560 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:41133
I20250810 13:59:04.113030 23627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:41133 every 8 connection(s)
I20250810 13:59:04.115847 23560 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:59:04.121057 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 23560
I20250810 13:59:04.122484 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:33745
--local_ip_for_outbound_sockets=127.19.27.65
--tserver_master_addrs=127.19.27.126:41133
--webserver_port=33051
--webserver_interface=127.19.27.65
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:59:04.125890 23628 sys_catalog.cc:263] Verifying existing consensus state
I20250810 13:59:04.131887 23628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap starting.
I20250810 13:59:04.147505 23628 log.cc:826] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:04.226192 23628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:04.227044 23628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Bootstrap complete.
I20250810 13:59:04.248942 23628 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:59:04.251010 23628 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d627fde845446419ace1c1815205f13, State: Initialized, Role: FOLLOWER
I20250810 13:59:04.251726 23628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:59:04.252260 23628 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:59:04.252511 23628 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:59:04.252784 23628 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 5 FOLLOWER]: Advancing to term 6
I20250810 13:59:04.257944 23628 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:59:04.258596 23628 leader_election.cc:304] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d627fde845446419ace1c1815205f13; no voters:
I20250810 13:59:04.260730 23628 leader_election.cc:290] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250810 13:59:04.261150 23632 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 6 FOLLOWER]: Leader election won for term 6
I20250810 13:59:04.264321 23632 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [term 6 LEADER]: Becoming Leader. State: Replica: 2d627fde845446419ace1c1815205f13, State: Running, Role: LEADER
I20250810 13:59:04.265211 23632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } }
I20250810 13:59:04.265746 23628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:59:04.276513 23633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:59:04.277251 23633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:59:04.275578 23634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d627fde845446419ace1c1815205f13. Latest consensus state: current_term: 6 leader_uuid: "2d627fde845446419ace1c1815205f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d627fde845446419ace1c1815205f13" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 41133 } } }
I20250810 13:59:04.280638 23634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13 [sys.catalog]: This master's current role is: LEADER
I20250810 13:59:04.290407 23639 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:59:04.305341 23639 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c377de843138483383111d66f8ca28bd]
I20250810 13:59:04.307407 23639 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=f67742e503c542d78e293dedb21fde69]
I20250810 13:59:04.309706 23639 catalog_manager.cc:671] Loaded metadata for table TestTable [id=fac60be6159740ab99a1e06daa3fc5c0]
I20250810 13:59:04.319164 23639 tablet_loader.cc:96] loaded metadata for tablet 0511d933a1a643ec9d645a015855fd42 (table TestTable1 [id=c377de843138483383111d66f8ca28bd])
I20250810 13:59:04.320708 23639 tablet_loader.cc:96] loaded metadata for tablet 34b81604d58e46fbbe8cc3cf164b8bdc (table TestTable2 [id=f67742e503c542d78e293dedb21fde69])
I20250810 13:59:04.321974 23639 tablet_loader.cc:96] loaded metadata for tablet 9056c7636f9543a0845e070fb56d24a2 (table TestTable [id=fac60be6159740ab99a1e06daa3fc5c0])
I20250810 13:59:04.323303 23639 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:59:04.328949 23639 catalog_manager.cc:1261] Loaded cluster ID: d97a81b24ea84adeb0c23ddf0223b00c
I20250810 13:59:04.329363 23639 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:59:04.334551 23650 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250810 13:59:04.338286 23639 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:59:04.344328 23639 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2d627fde845446419ace1c1815205f13: Loaded TSK: 0
I20250810 13:59:04.346022 23639 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 13:59:04.482558 23630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:04.483115 23630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:04.483886 23630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:04.514891 23630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:04.515800 23630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:59:04.550365 23630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:33745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=33051
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:04.551672 23630 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:04.553264 23630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:04.565858 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:04.567662 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:05.969035 23655 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23630
W20250810 13:59:06.090342 23630 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.523s user 0.485s sys 1.018s
W20250810 13:59:06.090809 23630 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.524s user 0.486s sys 1.019s
W20250810 13:59:06.093329 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:06.095497 23658 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1528 milliseconds
I20250810 13:59:06.095520 23630 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:06.096882 23630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:06.099083 23630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:06.100446 23630 hybrid_clock.cc:648] HybridClock initialized: now 1754834346100417 us; error 42 us; skew 500 ppm
I20250810 13:59:06.101231 23630 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:06.108367 23630 webserver.cc:489] Webserver started at http://127.19.27.65:33051/ using document root <none> and password file <none>
I20250810 13:59:06.109436 23630 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:06.109671 23630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:06.117810 23630 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250810 13:59:06.123337 23666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:06.124472 23630 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250810 13:59:06.124794 23630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4709b59128be4d74a9c65521d206ef95"
format_stamp: "Formatted at 2025-08-10 13:58:34 on dist-test-slave-k4vt"
I20250810 13:59:06.126664 23630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:06.176422 23630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:06.177973 23630 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:06.178442 23630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:06.181545 23630 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:06.188388 23673 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250810 13:59:06.209009 23630 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250810 13:59:06.209357 23630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.023s user 0.003s sys 0.000s
I20250810 13:59:06.209715 23630 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250810 13:59:06.218096 23673 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:59:06.229161 23630 ts_tablet_manager.cc:610] Registered 3 tablets
I20250810 13:59:06.229465 23630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.020s user 0.016s sys 0.003s
I20250810 13:59:06.280020 23673 log.cc:826] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:06.413707 23673 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:06.415077 23673 tablet_bootstrap.cc:492] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Bootstrap complete.
I20250810 13:59:06.417198 23673 ts_tablet_manager.cc:1397] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.199s user 0.145s sys 0.051s
I20250810 13:59:06.443346 23630 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:33745
I20250810 13:59:06.443493 23780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:33745 every 8 connection(s)
I20250810 13:59:06.444216 23673 raft_consensus.cc:357] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:59:06.446995 23630 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:59:06.447202 23673 raft_consensus.cc:738] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: FOLLOWER
I20250810 13:59:06.448145 23673 consensus_queue.cc:260] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:59:06.449000 23673 raft_consensus.cc:397] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:59:06.449453 23673 raft_consensus.cc:491] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:59:06.450018 23673 raft_consensus.cc:3058] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Advancing to term 3
I20250810 13:59:06.457244 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 23630
I20250810 13:59:06.459384 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:44701
--local_ip_for_outbound_sockets=127.19.27.66
--tserver_master_addrs=127.19.27.126:41133
--webserver_port=33899
--webserver_interface=127.19.27.66
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:59:06.462239 23673 raft_consensus.cc:513] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:59:06.464982 23673 leader_election.cc:304] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95; no voters:
I20250810 13:59:06.467371 23673 leader_election.cc:290] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250810 13:59:06.467938 23785 raft_consensus.cc:2802] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 3 FOLLOWER]: Leader election won for term 3
I20250810 13:59:06.476012 23785 raft_consensus.cc:695] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [term 3 LEADER]: Becoming Leader. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Running, Role: LEADER
I20250810 13:59:06.477324 23673 ts_tablet_manager.cc:1428] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.060s user 0.046s sys 0.010s
I20250810 13:59:06.478253 23673 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:59:06.477015 23785 consensus_queue.cc:237] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } }
I20250810 13:59:06.513182 23781 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:59:06.513787 23781 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:06.515151 23781 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:59:06.534446 23593 ts_manager.cc:194] Registered new tserver with Master: 4709b59128be4d74a9c65521d206ef95 (127.19.27.65:33745)
I20250810 13:59:06.541286 23593 catalog_manager.cc:5582] T 34b81604d58e46fbbe8cc3cf164b8bdc P 4709b59128be4d74a9c65521d206ef95 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } health_report { overall_health: HEALTHY } } }
I20250810 13:59:06.543100 23593 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 4709b59128be4d74a9c65521d206ef95 (127.19.27.65) added, VOTER e476695322144055a2a13384f8deaff4 (127.19.27.66) added, VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } } }
I20250810 13:59:06.634521 23593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:34983
I20250810 13:59:06.639986 23781 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:59:06.655529 23673 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:06.656435 23673 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Bootstrap complete.
I20250810 13:59:06.657969 23673 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.180s user 0.158s sys 0.015s
I20250810 13:59:06.660118 23673 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:06.660800 23673 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: FOLLOWER
I20250810 13:59:06.661470 23673 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:06.663312 23673 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250810 13:59:06.664039 23673 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Bootstrap starting.
I20250810 13:59:06.822675 23673 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:06.823560 23673 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Bootstrap complete.
I20250810 13:59:06.825104 23673 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Time spent bootstrapping tablet: real 0.161s user 0.144s sys 0.015s
I20250810 13:59:06.827214 23673 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:06.827904 23673 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Initialized, Role: FOLLOWER
I20250810 13:59:06.828524 23673 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:06.830343 23673 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250810 13:59:06.859107 23788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:06.859643 23788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:06.860190 23788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:06.892416 23788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:06.893265 23788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:59:06.929235 23788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:44701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=33899
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:06.930614 23788 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:06.932309 23788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:06.946069 23802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:08.089478 23808 raft_consensus.cc:491] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:59:08.090904 23808 raft_consensus.cc:513] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
W20250810 13:59:08.101044 23668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.27.66:44701: connect: Connection refused (error 111)
I20250810 13:59:08.103945 23808 leader_election.cc:290] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e476695322144055a2a13384f8deaff4 (127.19.27.66:44701), ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
W20250810 13:59:08.130237 23667 leader_election.cc:336] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111)
W20250810 13:59:08.132205 23668 leader_election.cc:336] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): Network error: Client connection negotiation failed: client connection to 127.19.27.66:44701: connect: Connection refused (error 111)
I20250810 13:59:08.132949 23668 leader_election.cc:304] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95; no voters: e476695322144055a2a13384f8deaff4, ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:59:08.134168 23808 raft_consensus.cc:2747] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250810 13:59:08.528365 23808 raft_consensus.cc:491] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:59:08.529078 23808 raft_consensus.cc:513] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:08.535208 23808 leader_election.cc:290] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069), e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
W20250810 13:59:08.540102 23667 leader_election.cc:336] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111)
W20250810 13:59:08.544641 23668 leader_election.cc:336] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): Network error: Client connection negotiation failed: client connection to 127.19.27.66:44701: connect: Connection refused (error 111)
I20250810 13:59:08.545357 23668 leader_election.cc:304] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95; no voters: e476695322144055a2a13384f8deaff4, ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:59:08.546404 23808 raft_consensus.cc:2747] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250810 13:59:08.349701 23801 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23788
W20250810 13:59:08.769580 23801 kernel_stack_watchdog.cc:198] Thread 23788 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:59:06.947147 23803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:08.771400 23788 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.824s user 0.661s sys 1.142s
W20250810 13:59:08.771836 23788 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.825s user 0.661s sys 1.143s
W20250810 13:59:08.773607 23805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:08.775883 23804 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1823 milliseconds
I20250810 13:59:08.775944 23788 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:08.777093 23788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:08.779211 23788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:08.780557 23788 hybrid_clock.cc:648] HybridClock initialized: now 1754834348780509 us; error 79 us; skew 500 ppm
I20250810 13:59:08.781320 23788 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:08.787446 23788 webserver.cc:489] Webserver started at http://127.19.27.66:33899/ using document root <none> and password file <none>
I20250810 13:59:08.788439 23788 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:08.788694 23788 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:08.796722 23788 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250810 13:59:08.801435 23816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:08.802486 23788 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250810 13:59:08.802810 23788 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e476695322144055a2a13384f8deaff4"
format_stamp: "Formatted at 2025-08-10 13:58:36 on dist-test-slave-k4vt"
I20250810 13:59:08.804847 23788 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:08.853916 23788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:08.855404 23788 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:08.855877 23788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:08.858462 23788 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:08.864282 23823 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250810 13:59:08.876731 23788 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250810 13:59:08.877043 23788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.000s
I20250810 13:59:08.877378 23788 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250810 13:59:08.882921 23823 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Bootstrap starting.
I20250810 13:59:08.886327 23788 ts_tablet_manager.cc:610] Registered 2 tablets
I20250810 13:59:08.886629 23788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20250810 13:59:08.955195 23823 log.cc:826] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:09.049912 23823 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:09.051003 23823 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Bootstrap complete.
I20250810 13:59:09.052480 23823 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Time spent bootstrapping tablet: real 0.170s user 0.131s sys 0.035s
I20250810 13:59:09.069293 23823 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:09.071894 23823 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e476695322144055a2a13384f8deaff4, State: Initialized, Role: FOLLOWER
I20250810 13:59:09.072836 23823 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:09.077107 23823 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4: Time spent starting tablet: real 0.024s user 0.023s sys 0.001s
I20250810 13:59:09.077863 23823 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap starting.
I20250810 13:59:09.102500 23788 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:44701
I20250810 13:59:09.102732 23931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:44701 every 8 connection(s)
I20250810 13:59:09.105989 23788 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:59:09.112608 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 23788
I20250810 13:59:09.114848 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:37069
--local_ip_for_outbound_sockets=127.19.27.67
--tserver_master_addrs=127.19.27.126:41133
--webserver_port=34405
--webserver_interface=127.19.27.67
--builtin_ntp_servers=127.19.27.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 13:59:09.160775 23932 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:59:09.161355 23932 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:09.162645 23932 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:59:09.167608 23593 ts_manager.cc:194] Registered new tserver with Master: e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
I20250810 13:59:09.172055 23593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:60929
I20250810 13:59:09.214361 23823 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:09.215080 23823 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Bootstrap complete.
I20250810 13:59:09.216292 23823 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Time spent bootstrapping tablet: real 0.139s user 0.106s sys 0.031s
I20250810 13:59:09.217919 23823 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:09.218361 23823 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e476695322144055a2a13384f8deaff4, State: Initialized, Role: FOLLOWER
I20250810 13:59:09.218834 23823 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:09.219858 23932 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:59:09.220424 23823 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20250810 13:59:09.458391 23937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:09.458956 23937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:09.459558 23937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:09.491732 23937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:09.492619 23937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:59:09.527819 23937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:37069
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=34405
--tserver_master_addrs=127.19.27.126:41133
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:09.529110 23937 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:09.530706 23937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:09.542439 23943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:10.090039 23949 raft_consensus.cc:491] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:59:10.090720 23949 raft_consensus.cc:513] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:10.093358 23949 leader_election.cc:290] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069), e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
I20250810 13:59:10.148345 23886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0511d933a1a643ec9d645a015855fd42" candidate_uuid: "4709b59128be4d74a9c65521d206ef95" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e476695322144055a2a13384f8deaff4" is_pre_election: true
I20250810 13:59:10.149665 23886 raft_consensus.cc:2466] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4709b59128be4d74a9c65521d206ef95 in term 2.
I20250810 13:59:10.152297 23668 leader_election.cc:304] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95, e476695322144055a2a13384f8deaff4; no voters:
I20250810 13:59:10.153566 23949 raft_consensus.cc:2802] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250810 13:59:10.154017 23949 raft_consensus.cc:491] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:59:10.154435 23949 raft_consensus.cc:3058] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Advancing to term 3
W20250810 13:59:10.156934 23667 leader_election.cc:336] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111)
I20250810 13:59:10.165933 23949 raft_consensus.cc:513] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:10.182327 23886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0511d933a1a643ec9d645a015855fd42" candidate_uuid: "4709b59128be4d74a9c65521d206ef95" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "e476695322144055a2a13384f8deaff4"
I20250810 13:59:10.183168 23886 raft_consensus.cc:3058] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Advancing to term 3
I20250810 13:59:10.195863 23949 leader_election.cc:290] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: Requested vote from peers ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069), e476695322144055a2a13384f8deaff4 (127.19.27.66:44701)
W20250810 13:59:10.195763 23667 leader_election.cc:336] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111)
I20250810 13:59:10.202024 23886 raft_consensus.cc:2466] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4709b59128be4d74a9c65521d206ef95 in term 3.
I20250810 13:59:10.203361 23668 leader_election.cc:304] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95, e476695322144055a2a13384f8deaff4; no voters: ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:59:10.204523 23949 raft_consensus.cc:2802] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 3 FOLLOWER]: Leader election won for term 3
I20250810 13:59:10.212316 23949 raft_consensus.cc:695] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [term 3 LEADER]: Becoming Leader. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Running, Role: LEADER
I20250810 13:59:10.213688 23949 consensus_queue.cc:237] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:10.220691 23954 raft_consensus.cc:491] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:59:10.221271 23954 raft_consensus.cc:513] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:10.224642 23954 leader_election.cc:290] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e476695322144055a2a13384f8deaff4 (127.19.27.66:44701), ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:59:10.224557 23886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9056c7636f9543a0845e070fb56d24a2" candidate_uuid: "4709b59128be4d74a9c65521d206ef95" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "e476695322144055a2a13384f8deaff4" is_pre_election: true
I20250810 13:59:10.226372 23886 raft_consensus.cc:2466] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4709b59128be4d74a9c65521d206ef95 in term 2.
I20250810 13:59:10.227492 23668 leader_election.cc:304] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95, e476695322144055a2a13384f8deaff4; no voters:
I20250810 13:59:10.228368 23954 raft_consensus.cc:2802] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250810 13:59:10.228766 23954 raft_consensus.cc:491] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:59:10.229142 23954 raft_consensus.cc:3058] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 2 FOLLOWER]: Advancing to term 3
W20250810 13:59:10.229717 23667 leader_election.cc:336] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111)
I20250810 13:59:10.236660 23954 raft_consensus.cc:513] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:10.250180 23954 leader_election.cc:290] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: Requested vote from peers e476695322144055a2a13384f8deaff4 (127.19.27.66:44701), ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:59:10.250568 23886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9056c7636f9543a0845e070fb56d24a2" candidate_uuid: "4709b59128be4d74a9c65521d206ef95" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "e476695322144055a2a13384f8deaff4"
I20250810 13:59:10.251711 23886 raft_consensus.cc:3058] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 2 FOLLOWER]: Advancing to term 3
W20250810 13:59:10.255348 23667 leader_election.cc:336] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111)
I20250810 13:59:10.272271 23593 catalog_manager.cc:5582] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: term changed from 2 to 3, leader changed from ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) to 4709b59128be4d74a9c65521d206ef95 (127.19.27.65). New cstate: current_term: 3 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250810 13:59:10.275348 23886 raft_consensus.cc:2466] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4709b59128be4d74a9c65521d206ef95 in term 3.
I20250810 13:59:10.277488 23668 leader_election.cc:304] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4709b59128be4d74a9c65521d206ef95, e476695322144055a2a13384f8deaff4; no voters: ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:59:10.279548 23955 raft_consensus.cc:2802] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 3 FOLLOWER]: Leader election won for term 3
I20250810 13:59:10.280369 23955 raft_consensus.cc:695] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 3 LEADER]: Becoming Leader. State: Replica: 4709b59128be4d74a9c65521d206ef95, State: Running, Role: LEADER
I20250810 13:59:10.281536 23955 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:10.311378 23593 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: term changed from 2 to 3, leader changed from <none> to 4709b59128be4d74a9c65521d206ef95 (127.19.27.65). New cstate: current_term: 3 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250810 13:59:10.613230 23886 raft_consensus.cc:1273] T 0511d933a1a643ec9d645a015855fd42 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Refusing update from remote peer 4709b59128be4d74a9c65521d206ef95: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250810 13:59:10.615665 23955 consensus_queue.cc:1035] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20250810 13:59:10.647161 23667 consensus_peers.cc:489] T 0511d933a1a643ec9d645a015855fd42 P 4709b59128be4d74a9c65521d206ef95 -> Peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Couldn't send request to peer ee3bf5aa8ab24508a06557a33da2cb5d. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250810 13:59:09.543905 23944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:10.844668 23946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:10.846518 23945 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1298 milliseconds
I20250810 13:59:10.846658 23937 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:10.847919 23937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:10.850713 23937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:10.852144 23937 hybrid_clock.cc:648] HybridClock initialized: now 1754834350852098 us; error 60 us; skew 500 ppm
I20250810 13:59:10.853003 23937 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:10.859263 23937 webserver.cc:489] Webserver started at http://127.19.27.67:34405/ using document root <none> and password file <none>
I20250810 13:59:10.860313 23937 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:10.860538 23937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:10.868722 23937 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250810 13:59:10.873585 23970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:10.874810 23937 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250810 13:59:10.875249 23937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ee3bf5aa8ab24508a06557a33da2cb5d"
format_stamp: "Formatted at 2025-08-10 13:58:38 on dist-test-slave-k4vt"
W20250810 13:59:10.877882 23667 consensus_peers.cc:489] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 -> Peer ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): Couldn't send request to peer ee3bf5aa8ab24508a06557a33da2cb5d. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.67:37069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 13:59:10.878096 23937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:10.888681 23886 raft_consensus.cc:1273] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Refusing update from remote peer 4709b59128be4d74a9c65521d206ef95: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250810 13:59:10.889847 23949 consensus_queue.cc:1035] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250810 13:59:10.922729 23736 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:10.928049 23886 raft_consensus.cc:1273] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Refusing update from remote peer 4709b59128be4d74a9c65521d206ef95: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250810 13:59:10.929311 23949 consensus_queue.cc:1035] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250810 13:59:10.935158 23954 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:59:10.937263 23886 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:59:10.940914 23937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:10.942930 23937 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:10.943140 23579 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9056c7636f9543a0845e070fb56d24a2 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 13:59:10.943579 23937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:10.947220 23937 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:10.947310 23593 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: config changed from index 13 to 15, VOTER ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67) evicted. New cstate: current_term: 3 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250810 13:59:10.955579 23981 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
W20250810 13:59:10.959184 23593 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 9056c7636f9543a0845e070fb56d24a2 on TS ee3bf5aa8ab24508a06557a33da2cb5d: Not found: failed to reset TS proxy: Could not find TS for UUID ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:59:10.962476 23736 consensus_queue.cc:237] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:10.964490 23954 raft_consensus.cc:2953] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER e476695322144055a2a13384f8deaff4 (127.19.27.66) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } }
I20250810 13:59:10.970335 23579 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9056c7636f9543a0845e070fb56d24a2 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 13:59:10.973754 23937 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250810 13:59:10.974025 23937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.000s sys 0.003s
I20250810 13:59:10.974318 23937 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250810 13:59:10.974738 23593 catalog_manager.cc:5582] T 9056c7636f9543a0845e070fb56d24a2 P 4709b59128be4d74a9c65521d206ef95 reported cstate change: config changed from index 15 to 16, VOTER e476695322144055a2a13384f8deaff4 (127.19.27.66) evicted. New cstate: current_term: 3 leader_uuid: "4709b59128be4d74a9c65521d206ef95" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250810 13:59:10.981225 23981 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap starting.
I20250810 13:59:10.987140 23937 ts_tablet_manager.cc:610] Registered 2 tablets
I20250810 13:59:10.987445 23937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.013s user 0.011s sys 0.000s
I20250810 13:59:10.998715 23866 tablet_service.cc:1515] Processing DeleteTablet for tablet 9056c7636f9543a0845e070fb56d24a2 with delete_type TABLET_DATA_TOMBSTONED (TS e476695322144055a2a13384f8deaff4 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:44914
I20250810 13:59:11.002120 23987 tablet_replica.cc:331] stopping tablet replica
I20250810 13:59:11.003163 23987 raft_consensus.cc:2241] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250810 13:59:11.003979 23987 raft_consensus.cc:2270] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250810 13:59:11.008073 23987 ts_tablet_manager.cc:1905] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250810 13:59:11.012503 23578 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 9056c7636f9543a0845e070fb56d24a2 on TS ee3bf5aa8ab24508a06557a33da2cb5d failed: Not found: failed to reset TS proxy: Could not find TS for UUID ee3bf5aa8ab24508a06557a33da2cb5d
I20250810 13:59:11.027278 23987 ts_tablet_manager.cc:1918] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250810 13:59:11.027920 23987 log.cc:1199] T 9056c7636f9543a0845e070fb56d24a2 P e476695322144055a2a13384f8deaff4: Deleting WAL directory at /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/wals/9056c7636f9543a0845e070fb56d24a2
I20250810 13:59:11.029910 23578 catalog_manager.cc:4928] TS e476695322144055a2a13384f8deaff4 (127.19.27.66:44701): tablet 9056c7636f9543a0845e070fb56d24a2 (table TestTable [id=fac60be6159740ab99a1e06daa3fc5c0]) successfully deleted
I20250810 13:59:11.045073 23981 log.cc:826] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:11.206022 23981 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:11.207329 23981 tablet_bootstrap.cc:492] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap complete.
I20250810 13:59:11.209559 23981 ts_tablet_manager.cc:1397] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent bootstrapping tablet: real 0.229s user 0.186s sys 0.037s
I20250810 13:59:11.223825 23937 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:37069
I20250810 13:59:11.223974 24092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:37069 every 8 connection(s)
I20250810 13:59:11.226859 23937 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:59:11.230036 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 23937
I20250810 13:59:11.232659 23981 raft_consensus.cc:357] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:11.239706 23981 raft_consensus.cc:738] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Initialized, Role: FOLLOWER
I20250810 13:59:11.240965 23981 consensus_queue.cc:260] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } } peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } }
I20250810 13:59:11.245636 23981 ts_tablet_manager.cc:1428] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent starting tablet: real 0.036s user 0.032s sys 0.000s
I20250810 13:59:11.246397 23981 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap starting.
I20250810 13:59:11.256945 24093 heartbeater.cc:344] Connected to a master server at 127.19.27.126:41133
I20250810 13:59:11.257416 24093 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:11.258420 24093 heartbeater.cc:507] Master 127.19.27.126:41133 requested a full tablet report, sending...
I20250810 13:59:11.262179 23593 ts_manager.cc:194] Registered new tserver with Master: ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069)
I20250810 13:59:11.265339 23593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:50077
I20250810 13:59:11.269781 24093 heartbeater.cc:499] Master 127.19.27.126:41133 was elected leader, sending a full tablet report...
I20250810 13:59:11.272536 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 13:59:11.276975 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250810 13:59:11.280385 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250810 13:59:11.346062 24028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9056c7636f9543a0845e070fb56d24a2 with delete_type TABLET_DATA_TOMBSTONED (TS ee3bf5aa8ab24508a06557a33da2cb5d not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:53758
W20250810 13:59:11.354470 23577 catalog_manager.cc:4908] TS ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): delete failed for tablet 9056c7636f9543a0845e070fb56d24a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9056c7636f9543a0845e070fb56d24a2 already in progress: opening tablet
I20250810 13:59:11.366529 23981 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 13:59:11.367242 23981 tablet_bootstrap.cc:492] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Bootstrap complete.
I20250810 13:59:11.368432 23981 ts_tablet_manager.cc:1397] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent bootstrapping tablet: real 0.122s user 0.107s sys 0.014s
I20250810 13:59:11.370007 23981 raft_consensus.cc:357] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:11.370450 23981 raft_consensus.cc:738] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3bf5aa8ab24508a06557a33da2cb5d, State: Initialized, Role: FOLLOWER
I20250810 13:59:11.370935 23981 consensus_queue.cc:260] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e476695322144055a2a13384f8deaff4" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 44701 } } peers { permanent_uuid: "4709b59128be4d74a9c65521d206ef95" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 33745 } attrs { promote: false } } peers { permanent_uuid: "ee3bf5aa8ab24508a06557a33da2cb5d" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 37069 } attrs { promote: false } }
I20250810 13:59:11.372342 23981 ts_tablet_manager.cc:1428] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250810 13:59:11.376691 24028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9056c7636f9543a0845e070fb56d24a2 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 13 (current committed config index is 16)) from {username='slave'} at 127.0.0.1:53758
I20250810 13:59:11.377679 24102 tablet_replica.cc:331] stopping tablet replica
I20250810 13:59:11.378304 24102 raft_consensus.cc:2241] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250810 13:59:11.378803 24102 raft_consensus.cc:2270] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250810 13:59:11.381937 24102 ts_tablet_manager.cc:1905] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 13:59:11.394029 24102 ts_tablet_manager.cc:1918] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250810 13:59:11.394379 24102 log.cc:1199] T 9056c7636f9543a0845e070fb56d24a2 P ee3bf5aa8ab24508a06557a33da2cb5d: Deleting WAL directory at /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/wals/9056c7636f9543a0845e070fb56d24a2
I20250810 13:59:11.395656 23577 catalog_manager.cc:4928] TS ee3bf5aa8ab24508a06557a33da2cb5d (127.19.27.67:37069): tablet 9056c7636f9543a0845e070fb56d24a2 (table TestTable [id=fac60be6159740ab99a1e06daa3fc5c0]) successfully deleted
I20250810 13:59:11.594411 24048 raft_consensus.cc:3058] T 0511d933a1a643ec9d645a015855fd42 P ee3bf5aa8ab24508a06557a33da2cb5d [term 2 FOLLOWER]: Advancing to term 3
W20250810 13:59:12.284623 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:13.288686 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:14.292184 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:15.296564 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:16.301540 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:17.305203 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:18.308765 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:19.312479 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:20.316116 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:21.319746 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:22.322940 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:23.326553 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:24.330044 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:25.334265 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:26.337996 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:27.341715 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:28.345399 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:29.349093 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 13:59:30.352871 19565 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9056c7636f9543a0845e070fb56d24a2: tablet_id: "9056c7636f9543a0845e070fb56d24a2" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250810 13:59:31.356110 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 23630
I20250810 13:59:31.384990 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 23788
I20250810 13:59:31.411164 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 23937
I20250810 13:59:31.436024 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 23560
2025-08-10T13:59:31Z chronyd exiting
I20250810 13:59:31.488762 19565 test_util.cc:183] -----------------------------------------------
I20250810 13:59:31.488977 19565 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754834243974047-19565-0
[ FAILED ] AdminCliTest.TestRebuildTables (61363 ms)
[----------] 5 tests from AdminCliTest (127449 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250810 13:59:31.492764 19565 test_util.cc:276] Using random seed: 928660355
I20250810 13:59:31.496922 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:59:31.497080 19565 ts_itest-base.cc:116] --------------
I20250810 13:59:31.497196 19565 ts_itest-base.cc:117] 5 tablet servers
I20250810 13:59:31.497310 19565 ts_itest-base.cc:118] 3 replicas per TS
I20250810 13:59:31.497407 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:59:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:59:31Z Disabled control of system clock
I20250810 13:59:31.542677 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:35557
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:35557
--raft_prepare_replacement_before_eviction=true with env {}
W20250810 13:59:31.850021 24127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:31.850577 24127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:31.850996 24127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:31.882429 24127 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 13:59:31.882772 24127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:59:31.882972 24127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:31.883163 24127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:59:31.883366 24127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:59:31.918535 24127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:35557
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:35557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:31.919826 24127 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:31.921382 24127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:31.933423 24133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:31.936928 24136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:31.933782 24134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:33.116956 24135 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1181 milliseconds
I20250810 13:59:33.117094 24127 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:33.118371 24127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:33.120996 24127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:33.122426 24127 hybrid_clock.cc:648] HybridClock initialized: now 1754834373122387 us; error 63 us; skew 500 ppm
I20250810 13:59:33.123240 24127 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:33.130020 24127 webserver.cc:489] Webserver started at http://127.19.27.126:38653/ using document root <none> and password file <none>
I20250810 13:59:33.131011 24127 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:33.131253 24127 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:33.131716 24127 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:33.136374 24127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "65c358a29a08435a96f598b725a0ede6"
format_stamp: "Formatted at 2025-08-10 13:59:33 on dist-test-slave-k4vt"
I20250810 13:59:33.137534 24127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "65c358a29a08435a96f598b725a0ede6"
format_stamp: "Formatted at 2025-08-10 13:59:33 on dist-test-slave-k4vt"
I20250810 13:59:33.145125 24127 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250810 13:59:33.150931 24143 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:33.152156 24127 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250810 13:59:33.152657 24127 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "65c358a29a08435a96f598b725a0ede6"
format_stamp: "Formatted at 2025-08-10 13:59:33 on dist-test-slave-k4vt"
I20250810 13:59:33.153029 24127 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:33.213644 24127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:33.215243 24127 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:33.215669 24127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:33.287206 24127 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:35557
I20250810 13:59:33.287279 24194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:35557 every 8 connection(s)
I20250810 13:59:33.290056 24127 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:59:33.295544 24195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:59:33.296412 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 24127
I20250810 13:59:33.296788 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:59:33.321957 24195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6: Bootstrap starting.
I20250810 13:59:33.327430 24195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6: Neither blocks nor log segments found. Creating new log.
I20250810 13:59:33.329237 24195 log.cc:826] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:33.333492 24195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6: No bootstrap required, opened a new log
I20250810 13:59:33.350790 24195 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c358a29a08435a96f598b725a0ede6" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 35557 } }
I20250810 13:59:33.351580 24195 raft_consensus.cc:383] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:59:33.351848 24195 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c358a29a08435a96f598b725a0ede6, State: Initialized, Role: FOLLOWER
I20250810 13:59:33.352478 24195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c358a29a08435a96f598b725a0ede6" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 35557 } }
I20250810 13:59:33.352949 24195 raft_consensus.cc:397] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:59:33.353176 24195 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:59:33.353438 24195 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:59:33.357589 24195 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c358a29a08435a96f598b725a0ede6" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 35557 } }
I20250810 13:59:33.358278 24195 leader_election.cc:304] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c358a29a08435a96f598b725a0ede6; no voters:
I20250810 13:59:33.359970 24195 leader_election.cc:290] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:59:33.360716 24200 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:59:33.362908 24200 raft_consensus.cc:695] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [term 1 LEADER]: Becoming Leader. State: Replica: 65c358a29a08435a96f598b725a0ede6, State: Running, Role: LEADER
I20250810 13:59:33.363721 24200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c358a29a08435a96f598b725a0ede6" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 35557 } }
I20250810 13:59:33.364773 24195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:59:33.374976 24202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c358a29a08435a96f598b725a0ede6. Latest consensus state: current_term: 1 leader_uuid: "65c358a29a08435a96f598b725a0ede6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c358a29a08435a96f598b725a0ede6" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 35557 } } }
I20250810 13:59:33.375926 24202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [sys.catalog]: This master's current role is: LEADER
I20250810 13:59:33.375675 24201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c358a29a08435a96f598b725a0ede6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c358a29a08435a96f598b725a0ede6" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 35557 } } }
I20250810 13:59:33.376359 24201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6 [sys.catalog]: This master's current role is: LEADER
I20250810 13:59:33.380115 24209 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:59:33.392112 24209 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:59:33.409720 24209 catalog_manager.cc:1349] Generated new cluster ID: 1023392f328a47b8aa84f193bf0ec715
I20250810 13:59:33.410039 24209 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:59:33.423509 24209 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:59:33.425122 24209 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:59:33.441322 24209 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 65c358a29a08435a96f598b725a0ede6: Generated new TSK 0
I20250810 13:59:33.442214 24209 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:59:33.461900 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--builtin_ntp_servers=127.19.27.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250810 13:59:33.797345 24219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:33.797876 24219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:33.798373 24219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:33.829890 24219 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 13:59:33.830313 24219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:33.831125 24219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:59:33.867872 24219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:33.869165 24219 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:33.870827 24219 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:33.884202 24225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:35.295652 24224 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 24219
W20250810 13:59:35.702612 24224 kernel_stack_watchdog.cc:198] Thread 24219 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 408ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:59:33.886560 24226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:35.703698 24227 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1819 milliseconds
W20250810 13:59:35.703939 24219 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.819s user 0.701s sys 1.019s
W20250810 13:59:35.704531 24219 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.820s user 0.701s sys 1.019s
W20250810 13:59:35.704727 24228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:35.704864 24219 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:35.708781 24219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:35.711021 24219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:35.712405 24219 hybrid_clock.cc:648] HybridClock initialized: now 1754834375712357 us; error 63 us; skew 500 ppm
I20250810 13:59:35.713276 24219 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:35.720137 24219 webserver.cc:489] Webserver started at http://127.19.27.65:45601/ using document root <none> and password file <none>
I20250810 13:59:35.721132 24219 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:35.721374 24219 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:35.721827 24219 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:35.726331 24219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "18c91a9e334a4fc2ad57dda4dbe136c4"
format_stamp: "Formatted at 2025-08-10 13:59:35 on dist-test-slave-k4vt"
I20250810 13:59:35.727502 24219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "18c91a9e334a4fc2ad57dda4dbe136c4"
format_stamp: "Formatted at 2025-08-10 13:59:35 on dist-test-slave-k4vt"
I20250810 13:59:35.735489 24219 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250810 13:59:35.741819 24236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:35.743008 24219 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 13:59:35.743412 24219 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "18c91a9e334a4fc2ad57dda4dbe136c4"
format_stamp: "Formatted at 2025-08-10 13:59:35 on dist-test-slave-k4vt"
I20250810 13:59:35.743750 24219 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:35.792650 24219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:35.794268 24219 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:35.794716 24219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:35.797879 24219 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:35.802209 24219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:59:35.802421 24219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:35.802711 24219 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:59:35.802868 24219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:35.977532 24219 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:46565
I20250810 13:59:35.977676 24348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:46565 every 8 connection(s)
I20250810 13:59:35.980399 24219 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:59:35.983314 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 24219
I20250810 13:59:35.984031 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:59:35.996044 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--builtin_ntp_servers=127.19.27.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 13:59:36.013597 24349 heartbeater.cc:344] Connected to a master server at 127.19.27.126:35557
I20250810 13:59:36.014117 24349 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:36.015216 24349 heartbeater.cc:507] Master 127.19.27.126:35557 requested a full tablet report, sending...
I20250810 13:59:36.017879 24160 ts_manager.cc:194] Registered new tserver with Master: 18c91a9e334a4fc2ad57dda4dbe136c4 (127.19.27.65:46565)
I20250810 13:59:36.020090 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:41333
W20250810 13:59:36.349684 24353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:36.350225 24353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:36.350731 24353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:36.382447 24353 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 13:59:36.382869 24353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:36.383671 24353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 13:59:36.419994 24353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:36.421357 24353 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:36.423010 24353 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:36.435338 24359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:37.023975 24349 heartbeater.cc:499] Master 127.19.27.126:35557 was elected leader, sending a full tablet report...
W20250810 13:59:36.436873 24360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:37.659165 24362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:37.661557 24361 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1221 milliseconds
I20250810 13:59:37.661700 24353 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:37.662909 24353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:37.665732 24353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:37.667289 24353 hybrid_clock.cc:648] HybridClock initialized: now 1754834377667231 us; error 109 us; skew 500 ppm
I20250810 13:59:37.668128 24353 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:37.674889 24353 webserver.cc:489] Webserver started at http://127.19.27.66:43013/ using document root <none> and password file <none>
I20250810 13:59:37.675884 24353 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:37.676088 24353 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:37.676529 24353 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:37.681123 24353 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "735f40de690e4b4e949496cde6212218"
format_stamp: "Formatted at 2025-08-10 13:59:37 on dist-test-slave-k4vt"
I20250810 13:59:37.682221 24353 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "735f40de690e4b4e949496cde6212218"
format_stamp: "Formatted at 2025-08-10 13:59:37 on dist-test-slave-k4vt"
I20250810 13:59:37.689563 24353 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250810 13:59:37.695521 24369 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:37.696617 24353 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250810 13:59:37.696962 24353 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "735f40de690e4b4e949496cde6212218"
format_stamp: "Formatted at 2025-08-10 13:59:37 on dist-test-slave-k4vt"
I20250810 13:59:37.697316 24353 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:37.751422 24353 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:37.752962 24353 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:37.753427 24353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:37.756049 24353 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:37.760350 24353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:59:37.760581 24353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:37.760846 24353 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:59:37.761018 24353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:37.898001 24353 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:43139
I20250810 13:59:37.898087 24481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:43139 every 8 connection(s)
I20250810 13:59:37.900822 24353 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 13:59:37.908542 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 24353
I20250810 13:59:37.909054 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 13:59:37.916132 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--builtin_ntp_servers=127.19.27.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 13:59:37.924105 24482 heartbeater.cc:344] Connected to a master server at 127.19.27.126:35557
I20250810 13:59:37.924641 24482 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:37.925920 24482 heartbeater.cc:507] Master 127.19.27.126:35557 requested a full tablet report, sending...
I20250810 13:59:37.928006 24160 ts_manager.cc:194] Registered new tserver with Master: 735f40de690e4b4e949496cde6212218 (127.19.27.66:43139)
I20250810 13:59:37.929193 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:58523
W20250810 13:59:38.214596 24486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:38.215092 24486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:38.215564 24486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:38.250029 24486 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 13:59:38.250411 24486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:38.251186 24486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 13:59:38.285229 24486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:38.286586 24486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:38.288292 24486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:38.301206 24492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:38.932148 24482 heartbeater.cc:499] Master 127.19.27.126:35557 was elected leader, sending a full tablet report...
W20250810 13:59:39.704607 24491 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 24486
W20250810 13:59:39.987818 24491 kernel_stack_watchdog.cc:198] Thread 24486 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:59:39.989080 24486 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.688s user 0.000s sys 0.002s
W20250810 13:59:39.989177 24494 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1686 milliseconds
W20250810 13:59:39.989491 24486 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.688s user 0.000s sys 0.002s
W20250810 13:59:38.303455 24493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:39.991497 24495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:39.991447 24486 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:39.994879 24486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:39.997089 24486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:39.998458 24486 hybrid_clock.cc:648] HybridClock initialized: now 1754834379998430 us; error 42 us; skew 500 ppm
I20250810 13:59:39.999276 24486 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:40.005541 24486 webserver.cc:489] Webserver started at http://127.19.27.67:39213/ using document root <none> and password file <none>
I20250810 13:59:40.006536 24486 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:40.006749 24486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:40.007167 24486 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:40.011514 24486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fb66358032bb4742a60613caefa22b4a"
format_stamp: "Formatted at 2025-08-10 13:59:40 on dist-test-slave-k4vt"
I20250810 13:59:40.012665 24486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fb66358032bb4742a60613caefa22b4a"
format_stamp: "Formatted at 2025-08-10 13:59:40 on dist-test-slave-k4vt"
I20250810 13:59:40.020259 24486 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 13:59:40.025903 24502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:40.027022 24486 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250810 13:59:40.027354 24486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb66358032bb4742a60613caefa22b4a"
format_stamp: "Formatted at 2025-08-10 13:59:40 on dist-test-slave-k4vt"
I20250810 13:59:40.027694 24486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:40.079082 24486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:40.080753 24486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:40.081341 24486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:40.084002 24486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:40.088317 24486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:59:40.088544 24486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:40.088798 24486 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:59:40.088970 24486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:40.226223 24486 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:39287
I20250810 13:59:40.226354 24614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:39287 every 8 connection(s)
I20250810 13:59:40.228874 24486 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 13:59:40.230206 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 24486
I20250810 13:59:40.230613 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 13:59:40.239393 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.68:0
--local_ip_for_outbound_sockets=127.19.27.68
--webserver_interface=127.19.27.68
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--builtin_ntp_servers=127.19.27.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 13:59:40.266799 24615 heartbeater.cc:344] Connected to a master server at 127.19.27.126:35557
I20250810 13:59:40.267236 24615 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:40.268355 24615 heartbeater.cc:507] Master 127.19.27.126:35557 requested a full tablet report, sending...
I20250810 13:59:40.270437 24160 ts_manager.cc:194] Registered new tserver with Master: fb66358032bb4742a60613caefa22b4a (127.19.27.67:39287)
I20250810 13:59:40.271726 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:47837
W20250810 13:59:40.554174 24619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:40.554687 24619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:40.555171 24619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:40.586527 24619 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 13:59:40.586936 24619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:40.587746 24619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.68
I20250810 13:59:40.622738 24619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.27.68
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.68
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:40.624179 24619 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:40.625774 24619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:40.637666 24625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:41.274931 24615 heartbeater.cc:499] Master 127.19.27.126:35557 was elected leader, sending a full tablet report...
W20250810 13:59:40.638350 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:42.047751 24624 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 24619
W20250810 13:59:42.410822 24624 kernel_stack_watchdog.cc:198] Thread 24619 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 403ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:59:42.417050 24619 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.779s user 0.632s sys 1.142s
W20250810 13:59:42.417511 24619 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.780s user 0.632s sys 1.142s
W20250810 13:59:42.417639 24627 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1778 milliseconds
W20250810 13:59:42.418884 24628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:42.418985 24619 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:42.420217 24619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:42.422271 24619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:42.423620 24619 hybrid_clock.cc:648] HybridClock initialized: now 1754834382423582 us; error 56 us; skew 500 ppm
I20250810 13:59:42.424469 24619 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:42.430987 24619 webserver.cc:489] Webserver started at http://127.19.27.68:43951/ using document root <none> and password file <none>
I20250810 13:59:42.432003 24619 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:42.432282 24619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:42.432778 24619 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:42.437244 24619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fb6003cad5da46a28e835a9135010214"
format_stamp: "Formatted at 2025-08-10 13:59:42 on dist-test-slave-k4vt"
I20250810 13:59:42.438369 24619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fb6003cad5da46a28e835a9135010214"
format_stamp: "Formatted at 2025-08-10 13:59:42 on dist-test-slave-k4vt"
I20250810 13:59:42.445806 24619 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250810 13:59:42.453227 24636 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:42.454738 24619 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250810 13:59:42.455169 24619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb6003cad5da46a28e835a9135010214"
format_stamp: "Formatted at 2025-08-10 13:59:42 on dist-test-slave-k4vt"
I20250810 13:59:42.455662 24619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:42.513429 24619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:42.515050 24619 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:42.515506 24619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:42.518263 24619 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:42.522902 24619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:59:42.523125 24619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:42.523401 24619 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:59:42.523564 24619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:42.667177 24619 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.68:45937
I20250810 13:59:42.667279 24748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.68:45937 every 8 connection(s)
I20250810 13:59:42.670063 24619 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250810 13:59:42.679064 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 24619
I20250810 13:59:42.679587 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250810 13:59:42.687485 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.69:0
--local_ip_for_outbound_sockets=127.19.27.69
--webserver_interface=127.19.27.69
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--builtin_ntp_servers=127.19.27.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 13:59:42.700350 24749 heartbeater.cc:344] Connected to a master server at 127.19.27.126:35557
I20250810 13:59:42.700819 24749 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:42.701964 24749 heartbeater.cc:507] Master 127.19.27.126:35557 requested a full tablet report, sending...
I20250810 13:59:42.704344 24160 ts_manager.cc:194] Registered new tserver with Master: fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937)
I20250810 13:59:42.705694 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.68:50245
W20250810 13:59:42.997386 24753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:42.997926 24753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:42.998431 24753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:43.029704 24753 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 13:59:43.030141 24753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:43.030933 24753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.69
I20250810 13:59:43.067199 24753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.27.69
--webserver_port=0
--tserver_master_addrs=127.19.27.126:35557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.69
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:43.068567 24753 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:43.070200 24753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:43.082710 24759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:43.709080 24749 heartbeater.cc:499] Master 127.19.27.126:35557 was elected leader, sending a full tablet report...
W20250810 13:59:44.486968 24758 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 24753
W20250810 13:59:44.558082 24758 kernel_stack_watchdog.cc:198] Thread 24753 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 13:59:43.083916 24760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:44.558928 24753 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.476s user 0.537s sys 0.931s
W20250810 13:59:44.559418 24761 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1474 milliseconds
W20250810 13:59:44.559422 24753 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.477s user 0.537s sys 0.932s
W20250810 13:59:44.561067 24762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 13:59:44.561116 24753 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:44.564576 24753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:44.566689 24753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:44.568030 24753 hybrid_clock.cc:648] HybridClock initialized: now 1754834384567996 us; error 51 us; skew 500 ppm
I20250810 13:59:44.568868 24753 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:44.574694 24753 webserver.cc:489] Webserver started at http://127.19.27.69:45229/ using document root <none> and password file <none>
I20250810 13:59:44.575659 24753 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:44.575901 24753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:44.576334 24753 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:44.580791 24753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "103ece953630450d90f7f92c0833bf0d"
format_stamp: "Formatted at 2025-08-10 13:59:44 on dist-test-slave-k4vt"
I20250810 13:59:44.581933 24753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "103ece953630450d90f7f92c0833bf0d"
format_stamp: "Formatted at 2025-08-10 13:59:44 on dist-test-slave-k4vt"
I20250810 13:59:44.589123 24753 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250810 13:59:44.594887 24771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:44.595945 24753 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250810 13:59:44.596256 24753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "103ece953630450d90f7f92c0833bf0d"
format_stamp: "Formatted at 2025-08-10 13:59:44 on dist-test-slave-k4vt"
I20250810 13:59:44.596614 24753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:44.649287 24753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:44.650748 24753 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:44.651183 24753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:44.653658 24753 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:44.657781 24753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:59:44.658000 24753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:44.658241 24753 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:59:44.658401 24753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:44.794970 24753 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.69:44799
I20250810 13:59:44.795045 24883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.69:44799 every 8 connection(s)
I20250810 13:59:44.797852 24753 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250810 13:59:44.803861 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 24753
I20250810 13:59:44.804538 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250810 13:59:44.819273 24884 heartbeater.cc:344] Connected to a master server at 127.19.27.126:35557
I20250810 13:59:44.819692 24884 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:44.820722 24884 heartbeater.cc:507] Master 127.19.27.126:35557 requested a full tablet report, sending...
I20250810 13:59:44.822952 24160 ts_manager.cc:194] Registered new tserver with Master: 103ece953630450d90f7f92c0833bf0d (127.19.27.69:44799)
I20250810 13:59:44.824517 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.69:43079
I20250810 13:59:44.826443 19565 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250810 13:59:44.863878 24159 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37040:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250810 13:59:44.946256 24417 tablet_service.cc:1468] Processing CreateTablet for tablet 1836a459c5e145c8ac852cbb14fb9a11 (DEFAULT_TABLE table=TestTable [id=8cd75214883941aebde80ff584a55ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:59:44.947710 24684 tablet_service.cc:1468] Processing CreateTablet for tablet 1836a459c5e145c8ac852cbb14fb9a11 (DEFAULT_TABLE table=TestTable [id=8cd75214883941aebde80ff584a55ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:59:44.948273 24417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1836a459c5e145c8ac852cbb14fb9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:59:44.949144 24684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1836a459c5e145c8ac852cbb14fb9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:59:44.949250 24819 tablet_service.cc:1468] Processing CreateTablet for tablet 1836a459c5e145c8ac852cbb14fb9a11 (DEFAULT_TABLE table=TestTable [id=8cd75214883941aebde80ff584a55ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:59:44.951161 24819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1836a459c5e145c8ac852cbb14fb9a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:59:44.974532 24903 tablet_bootstrap.cc:492] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: Bootstrap starting.
I20250810 13:59:44.976475 24904 tablet_bootstrap.cc:492] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: Bootstrap starting.
I20250810 13:59:44.983458 24903 tablet_bootstrap.cc:654] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: Neither blocks nor log segments found. Creating new log.
I20250810 13:59:44.983654 24904 tablet_bootstrap.cc:654] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: Neither blocks nor log segments found. Creating new log.
I20250810 13:59:44.985311 24903 log.cc:826] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:44.985824 24904 log.cc:826] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:44.992677 24904 tablet_bootstrap.cc:492] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: No bootstrap required, opened a new log
I20250810 13:59:44.992672 24907 tablet_bootstrap.cc:492] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: Bootstrap starting.
I20250810 13:59:44.993331 24904 ts_tablet_manager.cc:1397] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20250810 13:59:44.993695 24903 tablet_bootstrap.cc:492] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: No bootstrap required, opened a new log
I20250810 13:59:44.994091 24903 ts_tablet_manager.cc:1397] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.002s
I20250810 13:59:45.000388 24907 tablet_bootstrap.cc:654] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: Neither blocks nor log segments found. Creating new log.
I20250810 13:59:45.002847 24907 log.cc:826] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:45.009860 24907 tablet_bootstrap.cc:492] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: No bootstrap required, opened a new log
I20250810 13:59:45.010416 24907 ts_tablet_manager.cc:1397] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250810 13:59:45.013402 24903 raft_consensus.cc:357] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.014086 24903 raft_consensus.cc:383] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:59:45.014349 24903 raft_consensus.cc:738] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6003cad5da46a28e835a9135010214, State: Initialized, Role: FOLLOWER
I20250810 13:59:45.015111 24903 consensus_queue.cc:260] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.019086 24903 ts_tablet_manager.cc:1428] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: Time spent starting tablet: real 0.025s user 0.018s sys 0.006s
I20250810 13:59:45.020179 24904 raft_consensus.cc:357] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.021116 24904 raft_consensus.cc:383] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:59:45.021430 24904 raft_consensus.cc:738] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103ece953630450d90f7f92c0833bf0d, State: Initialized, Role: FOLLOWER
I20250810 13:59:45.022313 24904 consensus_queue.cc:260] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.025547 24884 heartbeater.cc:499] Master 127.19.27.126:35557 was elected leader, sending a full tablet report...
I20250810 13:59:45.027066 24904 ts_tablet_manager.cc:1428] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: Time spent starting tablet: real 0.033s user 0.032s sys 0.000s
I20250810 13:59:45.036464 24907 raft_consensus.cc:357] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.037174 24907 raft_consensus.cc:383] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:59:45.037388 24907 raft_consensus.cc:738] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735f40de690e4b4e949496cde6212218, State: Initialized, Role: FOLLOWER
I20250810 13:59:45.038362 24907 consensus_queue.cc:260] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.041342 24907 ts_tablet_manager.cc:1428] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: Time spent starting tablet: real 0.031s user 0.014s sys 0.014s
W20250810 13:59:45.052495 24885 tablet.cc:2378] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 13:59:45.172551 24483 tablet.cc:2378] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 13:59:45.178617 24750 tablet.cc:2378] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 13:59:45.341248 24911 raft_consensus.cc:491] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:59:45.341902 24910 raft_consensus.cc:491] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 13:59:45.341892 24911 raft_consensus.cc:513] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.342468 24910 raft_consensus.cc:513] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.345391 24911 leader_election.cc:290] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937), 103ece953630450d90f7f92c0833bf0d (127.19.27.69:44799)
I20250810 13:59:45.346072 24910 leader_election.cc:290] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937), 735f40de690e4b4e949496cde6212218 (127.19.27.66:43139)
I20250810 13:59:45.367192 24704 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "735f40de690e4b4e949496cde6212218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6003cad5da46a28e835a9135010214" is_pre_election: true
I20250810 13:59:45.368201 24704 raft_consensus.cc:2466] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 735f40de690e4b4e949496cde6212218 in term 0.
I20250810 13:59:45.369678 24370 leader_election.cc:304] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 735f40de690e4b4e949496cde6212218, fb6003cad5da46a28e835a9135010214; no voters:
I20250810 13:59:45.370625 24911 raft_consensus.cc:2802] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 13:59:45.371013 24911 raft_consensus.cc:491] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:59:45.371326 24911 raft_consensus.cc:3058] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:59:45.372711 24437 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "103ece953630450d90f7f92c0833bf0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "735f40de690e4b4e949496cde6212218" is_pre_election: true
I20250810 13:59:45.375763 24839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "735f40de690e4b4e949496cde6212218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103ece953630450d90f7f92c0833bf0d" is_pre_election: true
I20250810 13:59:45.375836 24704 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "103ece953630450d90f7f92c0833bf0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6003cad5da46a28e835a9135010214" is_pre_election: true
I20250810 13:59:45.376402 24839 raft_consensus.cc:2466] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 735f40de690e4b4e949496cde6212218 in term 0.
I20250810 13:59:45.376417 24704 raft_consensus.cc:2466] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103ece953630450d90f7f92c0833bf0d in term 0.
I20250810 13:59:45.377529 24772 leader_election.cc:304] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103ece953630450d90f7f92c0833bf0d, fb6003cad5da46a28e835a9135010214; no voters:
I20250810 13:59:45.378238 24910 raft_consensus.cc:2802] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 13:59:45.378494 24910 raft_consensus.cc:491] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 13:59:45.378749 24910 raft_consensus.cc:3058] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:59:45.378515 24911 raft_consensus.cc:513] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.379565 24437 raft_consensus.cc:2391] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 103ece953630450d90f7f92c0833bf0d in current term 1: Already voted for candidate 735f40de690e4b4e949496cde6212218 in this term.
I20250810 13:59:45.381302 24911 leader_election.cc:290] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [CANDIDATE]: Term 1 election: Requested vote from peers fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937), 103ece953630450d90f7f92c0833bf0d (127.19.27.69:44799)
I20250810 13:59:45.381534 24839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "735f40de690e4b4e949496cde6212218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103ece953630450d90f7f92c0833bf0d"
I20250810 13:59:45.382406 24704 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "735f40de690e4b4e949496cde6212218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6003cad5da46a28e835a9135010214"
I20250810 13:59:45.382979 24704 raft_consensus.cc:3058] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:59:45.384224 24910 raft_consensus.cc:513] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.385208 24839 raft_consensus.cc:2391] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 735f40de690e4b4e949496cde6212218 in current term 1: Already voted for candidate 103ece953630450d90f7f92c0833bf0d in this term.
I20250810 13:59:45.386528 24910 leader_election.cc:290] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [CANDIDATE]: Term 1 election: Requested vote from peers fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937), 735f40de690e4b4e949496cde6212218 (127.19.27.66:43139)
I20250810 13:59:45.388002 24437 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "103ece953630450d90f7f92c0833bf0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "735f40de690e4b4e949496cde6212218"
I20250810 13:59:45.388260 24703 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1836a459c5e145c8ac852cbb14fb9a11" candidate_uuid: "103ece953630450d90f7f92c0833bf0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6003cad5da46a28e835a9135010214"
I20250810 13:59:45.388823 24437 raft_consensus.cc:2391] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 103ece953630450d90f7f92c0833bf0d in current term 1: Already voted for candidate 735f40de690e4b4e949496cde6212218 in this term.
I20250810 13:59:45.389822 24772 leader_election.cc:304] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 103ece953630450d90f7f92c0833bf0d; no voters: 735f40de690e4b4e949496cde6212218, fb6003cad5da46a28e835a9135010214
I20250810 13:59:45.390951 24910 raft_consensus.cc:2747] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250810 13:59:45.391261 24704 raft_consensus.cc:2466] T 1836a459c5e145c8ac852cbb14fb9a11 P fb6003cad5da46a28e835a9135010214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 735f40de690e4b4e949496cde6212218 in term 1.
I20250810 13:59:45.392097 24370 leader_election.cc:304] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 735f40de690e4b4e949496cde6212218, fb6003cad5da46a28e835a9135010214; no voters: 103ece953630450d90f7f92c0833bf0d
I20250810 13:59:45.392689 24911 raft_consensus.cc:2802] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:59:45.394323 24911 raft_consensus.cc:695] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [term 1 LEADER]: Becoming Leader. State: Replica: 735f40de690e4b4e949496cde6212218, State: Running, Role: LEADER
I20250810 13:59:45.395073 24911 consensus_queue.cc:237] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } }
I20250810 13:59:45.404672 24160 catalog_manager.cc:5582] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 reported cstate change: term changed from 0 to 1, leader changed from <none> to 735f40de690e4b4e949496cde6212218 (127.19.27.66). New cstate: current_term: 1 leader_uuid: "735f40de690e4b4e949496cde6212218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb6003cad5da46a28e835a9135010214" member_type: VOTER last_known_addr { host: "127.19.27.68" port: 45937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "735f40de690e4b4e949496cde6212218" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 43139 } health_report { overall_health: HEALTHY } } }
I20250810 13:59:45.524883 19565 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250810 13:59:45.528156 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 735f40de690e4b4e949496cde6212218 to finish bootstrapping
I20250810 13:59:45.539958 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fb6003cad5da46a28e835a9135010214 to finish bootstrapping
I20250810 13:59:45.550818 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 103ece953630450d90f7f92c0833bf0d to finish bootstrapping
I20250810 13:59:45.561610 19565 test_util.cc:276] Using random seed: 942729203
I20250810 13:59:45.586006 19565 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250810 13:59:45.586881 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 24619
I20250810 13:59:45.620039 24839 raft_consensus.cc:1273] T 1836a459c5e145c8ac852cbb14fb9a11 P 103ece953630450d90f7f92c0833bf0d [term 1 FOLLOWER]: Refusing update from remote peer 735f40de690e4b4e949496cde6212218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250810 13:59:45.620494 24370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.27.68:45937: connect: Connection refused (error 111)
I20250810 13:59:45.621698 24911 consensus_queue.cc:1035] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103ece953630450d90f7f92c0833bf0d" member_type: VOTER last_known_addr { host: "127.19.27.69" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250810 13:59:45.628073 24370 consensus_peers.cc:489] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 -> Peer fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937): Couldn't send request to peer fb6003cad5da46a28e835a9135010214. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.68:45937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 13:59:45.639577 24928 mvcc.cc:204] Tried to move back new op lower bound from 7187801643481587712 to 7187801642585141248. Current Snapshot: MvccSnapshot[applied={T|T < 7187801643481587712}]
W20250810 13:59:47.139858 24370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.27.68:45937, user_credentials={real_user=slave}} blocked reactor thread for 47243.8us
W20250810 13:59:48.115082 24370 consensus_peers.cc:489] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 -> Peer fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937): Couldn't send request to peer fb6003cad5da46a28e835a9135010214. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.68:45937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250810 13:59:48.457381 24819 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 13:59:48.462718 24284 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 13:59:48.493137 24417 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:59:48.536841 24550 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250810 13:59:49.850510 24191 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 24128
W20250810 13:59:49.851231 24191 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1f60 after lost signal to thread 24194
I20250810 13:59:50.446280 24819 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 13:59:50.477790 24417 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 13:59:50.490876 24550 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 13:59:50.513737 24284 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250810 13:59:50.819443 24370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.27.68:45937: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250810 13:59:50.861282 24370 consensus_peers.cc:489] T 1836a459c5e145c8ac852cbb14fb9a11 P 735f40de690e4b4e949496cde6212218 -> Peer fb6003cad5da46a28e835a9135010214 (127.19.27.68:45937): Couldn't send request to peer fb6003cad5da46a28e835a9135010214. Status: Network error: Client connection negotiation failed: client connection to 127.19.27.68:45937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250810 13:59:53.044173 24611 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac3c0 after lost signal to thread 24487
W20250810 13:59:53.045082 24611 debug-util.cc:398] Leaking SignalData structure 0x7b08000abd80 after lost signal to thread 24614
I20250810 13:59:53.134871 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 24219
I20250810 13:59:53.172101 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 24353
I20250810 13:59:53.214726 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 24486
I20250810 13:59:53.242928 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 24753
I20250810 13:59:53.280544 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 24127
2025-08-10T13:59:53Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (21846 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (21846 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250810 13:59:53.339322 19565 test_util.cc:276] Using random seed: 950506909
I20250810 13:59:53.343252 19565 ts_itest-base.cc:115] Starting cluster with:
I20250810 13:59:53.343406 19565 ts_itest-base.cc:116] --------------
I20250810 13:59:53.343519 19565 ts_itest-base.cc:117] 1 tablet servers
I20250810 13:59:53.343631 19565 ts_itest-base.cc:118] 1 replicas per TS
I20250810 13:59:53.343729 19565 ts_itest-base.cc:119] --------------
2025-08-10T13:59:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T13:59:53Z Disabled control of system clock
I20250810 13:59:53.388043 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:46703
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:46703 with env {}
W20250810 13:59:53.693493 25036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:53.694052 25036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:53.694461 25036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:53.725889 25036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 13:59:53.726187 25036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:53.726398 25036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 13:59:53.726593 25036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 13:59:53.762189 25036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:46703
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:46703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:53.763504 25036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:53.765141 25036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:53.776625 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:53.780850 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:53.777334 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:54.948318 25044 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 13:59:54.948487 25036 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:54.952816 25036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:54.956068 25036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:54.957515 25036 hybrid_clock.cc:648] HybridClock initialized: now 1754834394957426 us; error 98 us; skew 500 ppm
I20250810 13:59:54.958741 25036 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:54.965101 25036 webserver.cc:489] Webserver started at http://127.19.27.126:37655/ using document root <none> and password file <none>
I20250810 13:59:54.966020 25036 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:54.966261 25036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:54.966742 25036 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:54.971220 25036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c37cd8b1137b4796bc98035e32cd3ce7"
format_stamp: "Formatted at 2025-08-10 13:59:54 on dist-test-slave-k4vt"
I20250810 13:59:54.972347 25036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c37cd8b1137b4796bc98035e32cd3ce7"
format_stamp: "Formatted at 2025-08-10 13:59:54 on dist-test-slave-k4vt"
I20250810 13:59:54.979646 25036 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 13:59:54.985122 25053 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:54.986197 25036 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250810 13:59:54.986579 25036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c37cd8b1137b4796bc98035e32cd3ce7"
format_stamp: "Formatted at 2025-08-10 13:59:54 on dist-test-slave-k4vt"
I20250810 13:59:54.986930 25036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:55.041885 25036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:55.043408 25036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:55.043875 25036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:55.113219 25036 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:46703
I20250810 13:59:55.113252 25104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:46703 every 8 connection(s)
I20250810 13:59:55.115870 25036 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 13:59:55.120851 25105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:59:55.123997 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25036
I20250810 13:59:55.124615 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 13:59:55.144897 25105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7: Bootstrap starting.
I20250810 13:59:55.151010 25105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7: Neither blocks nor log segments found. Creating new log.
I20250810 13:59:55.152937 25105 log.cc:826] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:55.157188 25105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7: No bootstrap required, opened a new log
I20250810 13:59:55.173836 25105 raft_consensus.cc:357] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 46703 } }
I20250810 13:59:55.174501 25105 raft_consensus.cc:383] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:59:55.174738 25105 raft_consensus.cc:738] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37cd8b1137b4796bc98035e32cd3ce7, State: Initialized, Role: FOLLOWER
I20250810 13:59:55.175491 25105 consensus_queue.cc:260] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 46703 } }
I20250810 13:59:55.176088 25105 raft_consensus.cc:397] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:59:55.176345 25105 raft_consensus.cc:491] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:59:55.176646 25105 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:59:55.181077 25105 raft_consensus.cc:513] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 46703 } }
I20250810 13:59:55.181836 25105 leader_election.cc:304] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c37cd8b1137b4796bc98035e32cd3ce7; no voters:
I20250810 13:59:55.183667 25105 leader_election.cc:290] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:59:55.184490 25110 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:59:55.186945 25110 raft_consensus.cc:695] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [term 1 LEADER]: Becoming Leader. State: Replica: c37cd8b1137b4796bc98035e32cd3ce7, State: Running, Role: LEADER
I20250810 13:59:55.187861 25110 consensus_queue.cc:237] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 46703 } }
I20250810 13:59:55.188797 25105 sys_catalog.cc:564] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 13:59:55.200273 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c37cd8b1137b4796bc98035e32cd3ce7. Latest consensus state: current_term: 1 leader_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 46703 } } }
I20250810 13:59:55.200373 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37cd8b1137b4796bc98035e32cd3ce7" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 46703 } } }
I20250810 13:59:55.201069 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [sys.catalog]: This master's current role is: LEADER
I20250810 13:59:55.201138 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7 [sys.catalog]: This master's current role is: LEADER
I20250810 13:59:55.204352 25120 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 13:59:55.215122 25120 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 13:59:55.229861 25120 catalog_manager.cc:1349] Generated new cluster ID: 1e3bd46de20b4dd5944f4663c3318d86
I20250810 13:59:55.230108 25120 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 13:59:55.250487 25120 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 13:59:55.251932 25120 catalog_manager.cc:1506] Loading token signing keys...
I20250810 13:59:55.266922 25120 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c37cd8b1137b4796bc98035e32cd3ce7: Generated new TSK 0
I20250810 13:59:55.267867 25120 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 13:59:55.277058 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:46703
--builtin_ntp_servers=127.19.27.84:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 13:59:55.584760 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 13:59:55.585309 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 13:59:55.585799 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 13:59:55.615938 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 13:59:55.616773 25129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 13:59:55.651225 25129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:46703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 13:59:55.653074 25129 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 13:59:55.654630 25129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 13:59:55.667690 25135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:55.669194 25136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:57.223407 25138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 13:59:57.225704 25137 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1556 milliseconds
I20250810 13:59:57.225826 25129 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 13:59:57.227113 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 13:59:57.229280 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 13:59:57.230639 25129 hybrid_clock.cc:648] HybridClock initialized: now 1754834397230601 us; error 55 us; skew 500 ppm
I20250810 13:59:57.231412 25129 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 13:59:57.238075 25129 webserver.cc:489] Webserver started at http://127.19.27.65:45293/ using document root <none> and password file <none>
I20250810 13:59:57.238999 25129 fs_manager.cc:362] Metadata directory not provided
I20250810 13:59:57.239215 25129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 13:59:57.239671 25129 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 13:59:57.244704 25129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0ec1d7654f084af1abff634cebd7ce50"
format_stamp: "Formatted at 2025-08-10 13:59:57 on dist-test-slave-k4vt"
I20250810 13:59:57.245792 25129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0ec1d7654f084af1abff634cebd7ce50"
format_stamp: "Formatted at 2025-08-10 13:59:57 on dist-test-slave-k4vt"
I20250810 13:59:57.253139 25129 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250810 13:59:57.258764 25145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:57.259819 25129 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250810 13:59:57.260144 25129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0ec1d7654f084af1abff634cebd7ce50"
format_stamp: "Formatted at 2025-08-10 13:59:57 on dist-test-slave-k4vt"
I20250810 13:59:57.260473 25129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 13:59:57.309695 25129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 13:59:57.311182 25129 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 13:59:57.311635 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 13:59:57.314117 25129 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 13:59:57.318329 25129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 13:59:57.318563 25129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:57.318806 25129 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 13:59:57.318964 25129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 13:59:57.450125 25129 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:41799
I20250810 13:59:57.450250 25257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:41799 every 8 connection(s)
I20250810 13:59:57.452814 25129 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 13:59:57.462066 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25129
I20250810 13:59:57.462472 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754834243974047-19565-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 13:59:57.473600 25258 heartbeater.cc:344] Connected to a master server at 127.19.27.126:46703
I20250810 13:59:57.474004 25258 heartbeater.cc:461] Registering TS with master...
I20250810 13:59:57.475240 25258 heartbeater.cc:507] Master 127.19.27.126:46703 requested a full tablet report, sending...
I20250810 13:59:57.477705 25070 ts_manager.cc:194] Registered new tserver with Master: 0ec1d7654f084af1abff634cebd7ce50 (127.19.27.65:41799)
I20250810 13:59:57.479622 25070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:56699
I20250810 13:59:57.482718 19565 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250810 13:59:57.518864 25070 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40396:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250810 13:59:57.574679 25193 tablet_service.cc:1468] Processing CreateTablet for tablet 997cf27e9a224ae2ab67963093028bbb (DEFAULT_TABLE table=TestTable [id=05a2e422dede4fd1b9f0e9476f4f2278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 13:59:57.576299 25193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 997cf27e9a224ae2ab67963093028bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 13:59:57.595757 25273 tablet_bootstrap.cc:492] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50: Bootstrap starting.
I20250810 13:59:57.601572 25273 tablet_bootstrap.cc:654] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50: Neither blocks nor log segments found. Creating new log.
I20250810 13:59:57.603283 25273 log.cc:826] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50: Log is configured to *not* fsync() on all Append() calls
I20250810 13:59:57.607623 25273 tablet_bootstrap.cc:492] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50: No bootstrap required, opened a new log
I20250810 13:59:57.608078 25273 ts_tablet_manager.cc:1397] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20250810 13:59:57.624714 25273 raft_consensus.cc:357] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec1d7654f084af1abff634cebd7ce50" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41799 } }
I20250810 13:59:57.625272 25273 raft_consensus.cc:383] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 13:59:57.625530 25273 raft_consensus.cc:738] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec1d7654f084af1abff634cebd7ce50, State: Initialized, Role: FOLLOWER
I20250810 13:59:57.626202 25273 consensus_queue.cc:260] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec1d7654f084af1abff634cebd7ce50" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41799 } }
I20250810 13:59:57.626704 25273 raft_consensus.cc:397] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 13:59:57.626933 25273 raft_consensus.cc:491] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 13:59:57.627204 25273 raft_consensus.cc:3058] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 0 FOLLOWER]: Advancing to term 1
I20250810 13:59:57.631204 25273 raft_consensus.cc:513] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec1d7654f084af1abff634cebd7ce50" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41799 } }
I20250810 13:59:57.631919 25273 leader_election.cc:304] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec1d7654f084af1abff634cebd7ce50; no voters:
I20250810 13:59:57.633639 25273 leader_election.cc:290] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 13:59:57.634443 25275 raft_consensus.cc:2802] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 13:59:57.636474 25258 heartbeater.cc:499] Master 127.19.27.126:46703 was elected leader, sending a full tablet report...
I20250810 13:59:57.637115 25275 raft_consensus.cc:695] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec1d7654f084af1abff634cebd7ce50, State: Running, Role: LEADER
I20250810 13:59:57.637924 25273 ts_tablet_manager.cc:1428] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50: Time spent starting tablet: real 0.030s user 0.024s sys 0.005s
I20250810 13:59:57.637928 25275 consensus_queue.cc:237] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec1d7654f084af1abff634cebd7ce50" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41799 } }
I20250810 13:59:57.650444 25070 catalog_manager.cc:5582] T 997cf27e9a224ae2ab67963093028bbb P 0ec1d7654f084af1abff634cebd7ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec1d7654f084af1abff634cebd7ce50 (127.19.27.65). New cstate: current_term: 1 leader_uuid: "0ec1d7654f084af1abff634cebd7ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec1d7654f084af1abff634cebd7ce50" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 41799 } health_report { overall_health: HEALTHY } } }
I20250810 13:59:57.681102 19565 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250810 13:59:57.684217 19565 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0ec1d7654f084af1abff634cebd7ce50 to finish bootstrapping
I20250810 14:00:00.419981 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25129
I20250810 14:00:00.445572 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25036
2025-08-10T14:00:00Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7163 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7163 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250810 14:00:00.503414 19565 test_util.cc:276] Using random seed: 957671003
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-10T14:00:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T14:00:00Z Disabled control of system clock
I20250810 14:00:00.556879 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:45779
--webserver_interface=127.19.27.126
--webserver_port=0
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:45779 with env {}
W20250810 14:00:00.866036 25301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:00.866827 25301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:00.867324 25301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:00.898801 25301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 14:00:00.899134 25301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:00.899407 25301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 14:00:00.899652 25301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 14:00:00.935504 25301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:45779
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:00.936877 25301 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:00.938562 25301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:00.949473 25307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:00.950215 25308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:02.151412 25310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:02.154471 25309 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1203 milliseconds
I20250810 14:00:02.154599 25301 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 14:00:02.155915 25301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:02.158511 25301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:02.159915 25301 hybrid_clock.cc:648] HybridClock initialized: now 1754834402159877 us; error 61 us; skew 500 ppm
I20250810 14:00:02.160724 25301 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:02.167364 25301 webserver.cc:489] Webserver started at http://127.19.27.126:38703/ using document root <none> and password file <none>
I20250810 14:00:02.168346 25301 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:02.168583 25301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:02.169047 25301 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 14:00:02.173805 25301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/instance:
uuid: "a2e0584fe23b4feb8de7c337160a0db5"
format_stamp: "Formatted at 2025-08-10 14:00:02 on dist-test-slave-k4vt"
I20250810 14:00:02.175026 25301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal/instance:
uuid: "a2e0584fe23b4feb8de7c337160a0db5"
format_stamp: "Formatted at 2025-08-10 14:00:02 on dist-test-slave-k4vt"
I20250810 14:00:02.182592 25301 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 14:00:02.188485 25317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:02.189687 25301 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 14:00:02.190061 25301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
uuid: "a2e0584fe23b4feb8de7c337160a0db5"
format_stamp: "Formatted at 2025-08-10 14:00:02 on dist-test-slave-k4vt"
I20250810 14:00:02.190438 25301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:02.245052 25301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:02.246570 25301 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:02.247042 25301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:02.317416 25301 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:45779
I20250810 14:00:02.317482 25368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:45779 every 8 connection(s)
I20250810 14:00:02.320277 25301 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb
I20250810 14:00:02.325469 25369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:02.329311 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25301
I20250810 14:00:02.329782 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal/instance
I20250810 14:00:02.349372 25369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5: Bootstrap starting.
I20250810 14:00:02.355760 25369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:02.357841 25369 log.cc:826] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:02.362748 25369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5: No bootstrap required, opened a new log
I20250810 14:00:02.381745 25369 raft_consensus.cc:357] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e0584fe23b4feb8de7c337160a0db5" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:02.382431 25369 raft_consensus.cc:383] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:02.382647 25369 raft_consensus.cc:738] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e0584fe23b4feb8de7c337160a0db5, State: Initialized, Role: FOLLOWER
I20250810 14:00:02.383323 25369 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e0584fe23b4feb8de7c337160a0db5" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:02.383821 25369 raft_consensus.cc:397] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 14:00:02.384052 25369 raft_consensus.cc:491] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 14:00:02.384382 25369 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:02.389165 25369 raft_consensus.cc:513] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e0584fe23b4feb8de7c337160a0db5" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:02.389859 25369 leader_election.cc:304] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e0584fe23b4feb8de7c337160a0db5; no voters:
I20250810 14:00:02.391521 25369 leader_election.cc:290] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 14:00:02.392369 25374 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 14:00:02.394709 25374 raft_consensus.cc:695] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [term 1 LEADER]: Becoming Leader. State: Replica: a2e0584fe23b4feb8de7c337160a0db5, State: Running, Role: LEADER
I20250810 14:00:02.395676 25369 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 14:00:02.395560 25374 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e0584fe23b4feb8de7c337160a0db5" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:02.406950 25375 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e0584fe23b4feb8de7c337160a0db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e0584fe23b4feb8de7c337160a0db5" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } } }
I20250810 14:00:02.409322 25375 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [sys.catalog]: This master's current role is: LEADER
I20250810 14:00:02.408565 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e0584fe23b4feb8de7c337160a0db5. Latest consensus state: current_term: 1 leader_uuid: "a2e0584fe23b4feb8de7c337160a0db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e0584fe23b4feb8de7c337160a0db5" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } } }
I20250810 14:00:02.411377 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5 [sys.catalog]: This master's current role is: LEADER
I20250810 14:00:02.421708 25384 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 14:00:02.429633 25384 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 14:00:02.444552 25384 catalog_manager.cc:1349] Generated new cluster ID: fdba5080d4ab4e19bf555af1d3bfd165
I20250810 14:00:02.444892 25384 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 14:00:02.490820 25384 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 14:00:02.493041 25384 catalog_manager.cc:1506] Loading token signing keys...
I20250810 14:00:02.516661 25384 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a2e0584fe23b4feb8de7c337160a0db5: Generated new TSK 0
I20250810 14:00:02.517602 25384 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 14:00:02.534952 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:0
--local_ip_for_outbound_sockets=127.19.27.65
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:45779
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250810 14:00:02.842669 25393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:02.843186 25393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:02.843703 25393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:02.874825 25393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:02.875690 25393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 14:00:02.910728 25393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=0
--tserver_master_addrs=127.19.27.126:45779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:02.912333 25393 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:02.913905 25393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:02.926972 25399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:02.931604 25402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:02.928711 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:04.462028 25401 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1532 milliseconds
I20250810 14:00:04.462141 25393 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 14:00:04.463335 25393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:04.465731 25393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:04.467101 25393 hybrid_clock.cc:648] HybridClock initialized: now 1754834404467042 us; error 70 us; skew 500 ppm
I20250810 14:00:04.467932 25393 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:04.474081 25393 webserver.cc:489] Webserver started at http://127.19.27.65:46197/ using document root <none> and password file <none>
I20250810 14:00:04.475044 25393 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:04.475266 25393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:04.475762 25393 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 14:00:04.480072 25393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/instance:
uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1"
format_stamp: "Formatted at 2025-08-10 14:00:04 on dist-test-slave-k4vt"
I20250810 14:00:04.481186 25393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal/instance:
uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1"
format_stamp: "Formatted at 2025-08-10 14:00:04 on dist-test-slave-k4vt"
I20250810 14:00:04.488168 25393 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250810 14:00:04.493640 25409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:04.494570 25393 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250810 14:00:04.494895 25393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1"
format_stamp: "Formatted at 2025-08-10 14:00:04 on dist-test-slave-k4vt"
I20250810 14:00:04.495219 25393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:04.539276 25393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:04.540683 25393 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:04.541101 25393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:04.543473 25393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 14:00:04.547353 25393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 14:00:04.547573 25393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:04.547847 25393 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 14:00:04.548007 25393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:04.677953 25393 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:42565
I20250810 14:00:04.678056 25522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:42565 every 8 connection(s)
I20250810 14:00:04.680522 25393 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb
I20250810 14:00:04.685653 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25393
I20250810 14:00:04.686415 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal/instance
I20250810 14:00:04.693020 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:0
--local_ip_for_outbound_sockets=127.19.27.66
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:45779
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 14:00:04.702950 25523 heartbeater.cc:344] Connected to a master server at 127.19.27.126:45779
I20250810 14:00:04.703521 25523 heartbeater.cc:461] Registering TS with master...
I20250810 14:00:04.704906 25523 heartbeater.cc:507] Master 127.19.27.126:45779 requested a full tablet report, sending...
I20250810 14:00:04.707772 25334 ts_manager.cc:194] Registered new tserver with Master: d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565)
I20250810 14:00:04.709830 25334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:44785
W20250810 14:00:04.994415 25527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:04.994940 25527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:04.995429 25527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:05.028324 25527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:05.029318 25527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 14:00:05.063400 25527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=0
--tserver_master_addrs=127.19.27.126:45779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:05.064764 25527 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:05.066351 25527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:05.078081 25533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:05.714138 25523 heartbeater.cc:499] Master 127.19.27.126:45779 was elected leader, sending a full tablet report...
W20250810 14:00:05.078998 25534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:06.253720 25536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:06.256068 25535 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1172 milliseconds
I20250810 14:00:06.256243 25527 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 14:00:06.257431 25527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:06.260102 25527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:06.261533 25527 hybrid_clock.cc:648] HybridClock initialized: now 1754834406261438 us; error 100 us; skew 500 ppm
I20250810 14:00:06.262364 25527 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:06.269094 25527 webserver.cc:489] Webserver started at http://127.19.27.66:43743/ using document root <none> and password file <none>
I20250810 14:00:06.270032 25527 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:06.270226 25527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:06.270661 25527 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 14:00:06.275141 25527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/instance:
uuid: "4cb99619485548efa655fee15236b25a"
format_stamp: "Formatted at 2025-08-10 14:00:06 on dist-test-slave-k4vt"
I20250810 14:00:06.276289 25527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal/instance:
uuid: "4cb99619485548efa655fee15236b25a"
format_stamp: "Formatted at 2025-08-10 14:00:06 on dist-test-slave-k4vt"
I20250810 14:00:06.283185 25527 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250810 14:00:06.289021 25543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:06.290122 25527 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250810 14:00:06.290433 25527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
uuid: "4cb99619485548efa655fee15236b25a"
format_stamp: "Formatted at 2025-08-10 14:00:06 on dist-test-slave-k4vt"
I20250810 14:00:06.290764 25527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:06.356827 25527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:06.358217 25527 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:06.358615 25527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:06.361090 25527 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 14:00:06.365165 25527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 14:00:06.365362 25527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:06.365569 25527 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 14:00:06.365701 25527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:06.496433 25527 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:35313
I20250810 14:00:06.496558 25655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:35313 every 8 connection(s)
I20250810 14:00:06.498951 25527 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb
I20250810 14:00:06.508265 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25527
I20250810 14:00:06.508782 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal/instance
I20250810 14:00:06.515352 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:0
--local_ip_for_outbound_sockets=127.19.27.67
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:45779
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 14:00:06.521126 25656 heartbeater.cc:344] Connected to a master server at 127.19.27.126:45779
I20250810 14:00:06.521664 25656 heartbeater.cc:461] Registering TS with master...
I20250810 14:00:06.523022 25656 heartbeater.cc:507] Master 127.19.27.126:45779 requested a full tablet report, sending...
I20250810 14:00:06.525588 25334 ts_manager.cc:194] Registered new tserver with Master: 4cb99619485548efa655fee15236b25a (127.19.27.66:35313)
I20250810 14:00:06.526782 25334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:52151
W20250810 14:00:06.819429 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:06.819986 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:06.820482 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:06.851672 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:06.852602 25660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 14:00:06.886292 25660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=0
--tserver_master_addrs=127.19.27.126:45779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:06.887637 25660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:06.889200 25660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:06.900949 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:07.530144 25656 heartbeater.cc:499] Master 127.19.27.126:45779 was elected leader, sending a full tablet report...
W20250810 14:00:06.901667 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:08.200135 25668 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1298 milliseconds
W20250810 14:00:08.201675 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:08.204245 25660 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.304s user 0.386s sys 0.896s
W20250810 14:00:08.204622 25660 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.304s user 0.386s sys 0.896s
I20250810 14:00:08.204921 25660 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 14:00:08.206413 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:08.209357 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:08.210886 25660 hybrid_clock.cc:648] HybridClock initialized: now 1754834408210848 us; error 34 us; skew 500 ppm
I20250810 14:00:08.212143 25660 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:08.221338 25660 webserver.cc:489] Webserver started at http://127.19.27.67:35125/ using document root <none> and password file <none>
I20250810 14:00:08.222965 25660 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:08.223315 25660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:08.224015 25660 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 14:00:08.231519 25660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/instance:
uuid: "85ceea734f87464d8b1cb326b1a6c56f"
format_stamp: "Formatted at 2025-08-10 14:00:08 on dist-test-slave-k4vt"
I20250810 14:00:08.233289 25660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal/instance:
uuid: "85ceea734f87464d8b1cb326b1a6c56f"
format_stamp: "Formatted at 2025-08-10 14:00:08 on dist-test-slave-k4vt"
I20250810 14:00:08.244520 25660 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.006s
I20250810 14:00:08.252931 25676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:08.254257 25660 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250810 14:00:08.254719 25660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
uuid: "85ceea734f87464d8b1cb326b1a6c56f"
format_stamp: "Formatted at 2025-08-10 14:00:08 on dist-test-slave-k4vt"
I20250810 14:00:08.255204 25660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:08.359607 25660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:08.361100 25660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:08.361514 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:08.364069 25660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 14:00:08.368124 25660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 14:00:08.368340 25660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:08.368542 25660 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 14:00:08.368683 25660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:08.506484 25660 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:35679
I20250810 14:00:08.506594 25788 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:35679 every 8 connection(s)
I20250810 14:00:08.509096 25660 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb
I20250810 14:00:08.514894 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25660
I20250810 14:00:08.515482 19565 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal/instance
I20250810 14:00:08.535823 25789 heartbeater.cc:344] Connected to a master server at 127.19.27.126:45779
I20250810 14:00:08.536353 25789 heartbeater.cc:461] Registering TS with master...
I20250810 14:00:08.537564 25789 heartbeater.cc:507] Master 127.19.27.126:45779 requested a full tablet report, sending...
I20250810 14:00:08.539739 25334 ts_manager.cc:194] Registered new tserver with Master: 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
I20250810 14:00:08.540998 25334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:50365
I20250810 14:00:08.550123 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 14:00:08.580945 19565 test_util.cc:276] Using random seed: 965748535
I20250810 14:00:08.621495 25334 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37080:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250810 14:00:08.624011 25334 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 14:00:08.679996 25591 tablet_service.cc:1468] Processing CreateTablet for tablet f57feee9554e42678aededb7914ab7ba (DEFAULT_TABLE table=pre_rebuild [id=ecdc7522cf6e48d7b8e645c20b2df5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 14:00:08.682206 25591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57feee9554e42678aededb7914ab7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:08.682073 25458 tablet_service.cc:1468] Processing CreateTablet for tablet f57feee9554e42678aededb7914ab7ba (DEFAULT_TABLE table=pre_rebuild [id=ecdc7522cf6e48d7b8e645c20b2df5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 14:00:08.684173 25458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57feee9554e42678aededb7914ab7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:08.683861 25724 tablet_service.cc:1468] Processing CreateTablet for tablet f57feee9554e42678aededb7914ab7ba (DEFAULT_TABLE table=pre_rebuild [id=ecdc7522cf6e48d7b8e645c20b2df5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 14:00:08.685762 25724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57feee9554e42678aededb7914ab7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:08.717595 25815 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Bootstrap starting.
I20250810 14:00:08.718837 25814 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Bootstrap starting.
I20250810 14:00:08.719616 25813 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Bootstrap starting.
I20250810 14:00:08.726253 25815 tablet_bootstrap.cc:654] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:08.726792 25814 tablet_bootstrap.cc:654] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:08.728487 25813 tablet_bootstrap.cc:654] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:08.728992 25815 log.cc:826] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:08.729652 25814 log.cc:826] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:08.730804 25813 log.cc:826] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:08.738834 25814 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: No bootstrap required, opened a new log
I20250810 14:00:08.739271 25813 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: No bootstrap required, opened a new log
I20250810 14:00:08.739426 25814 ts_tablet_manager.cc:1397] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.006s
I20250810 14:00:08.739611 25815 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: No bootstrap required, opened a new log
I20250810 14:00:08.739699 25813 ts_tablet_manager.cc:1397] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.008s
I20250810 14:00:08.740180 25815 ts_tablet_manager.cc:1397] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.008s
I20250810 14:00:08.759056 25813 raft_consensus.cc:357] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.759821 25813 raft_consensus.cc:383] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:08.760097 25813 raft_consensus.cc:738] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb99619485548efa655fee15236b25a, State: Initialized, Role: FOLLOWER
I20250810 14:00:08.760860 25813 consensus_queue.cc:260] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.765143 25813 ts_tablet_manager.cc:1428] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Time spent starting tablet: real 0.025s user 0.022s sys 0.002s
I20250810 14:00:08.770079 25814 raft_consensus.cc:357] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.770081 25815 raft_consensus.cc:357] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.771112 25815 raft_consensus.cc:383] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:08.771107 25814 raft_consensus.cc:383] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:08.771450 25814 raft_consensus.cc:738] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d804e4e7c4e9457d86ab55cb9e9b25e1, State: Initialized, Role: FOLLOWER
I20250810 14:00:08.771548 25815 raft_consensus.cc:738] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ceea734f87464d8b1cb326b1a6c56f, State: Initialized, Role: FOLLOWER
I20250810 14:00:08.772454 25814 consensus_queue.cc:260] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.772574 25815 consensus_queue.cc:260] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.775667 25789 heartbeater.cc:499] Master 127.19.27.126:45779 was elected leader, sending a full tablet report...
I20250810 14:00:08.777313 25815 ts_tablet_manager.cc:1428] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Time spent starting tablet: real 0.037s user 0.029s sys 0.007s
I20250810 14:00:08.780357 25814 ts_tablet_manager.cc:1428] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Time spent starting tablet: real 0.041s user 0.039s sys 0.003s
I20250810 14:00:08.812731 25819 raft_consensus.cc:491] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 14:00:08.813231 25819 raft_consensus.cc:513] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.815521 25819 leader_election.cc:290] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565), 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
I20250810 14:00:08.829346 25744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ceea734f87464d8b1cb326b1a6c56f" is_pre_election: true
I20250810 14:00:08.829743 25478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" is_pre_election: true
I20250810 14:00:08.830302 25744 raft_consensus.cc:2466] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 0.
I20250810 14:00:08.830592 25478 raft_consensus.cc:2466] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 0.
I20250810 14:00:08.831683 25547 leader_election.cc:304] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb99619485548efa655fee15236b25a, 85ceea734f87464d8b1cb326b1a6c56f; no voters:
I20250810 14:00:08.832419 25819 raft_consensus.cc:2802] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 14:00:08.832687 25819 raft_consensus.cc:491] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 14:00:08.832963 25819 raft_consensus.cc:3058] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:08.837141 25819 raft_consensus.cc:513] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.838559 25819 leader_election.cc:290] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 election: Requested vote from peers d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565), 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
I20250810 14:00:08.839476 25478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1"
I20250810 14:00:08.839596 25744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ceea734f87464d8b1cb326b1a6c56f"
I20250810 14:00:08.840045 25478 raft_consensus.cc:3058] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:08.840137 25744 raft_consensus.cc:3058] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:08.847036 25478 raft_consensus.cc:2466] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 1.
I20250810 14:00:08.847040 25744 raft_consensus.cc:2466] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 1.
I20250810 14:00:08.848248 25545 leader_election.cc:304] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb99619485548efa655fee15236b25a, d804e4e7c4e9457d86ab55cb9e9b25e1; no voters:
I20250810 14:00:08.848974 25819 raft_consensus.cc:2802] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Leader election won for term 1
I20250810 14:00:08.850498 25819 raft_consensus.cc:695] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 LEADER]: Becoming Leader. State: Replica: 4cb99619485548efa655fee15236b25a, State: Running, Role: LEADER
I20250810 14:00:08.851356 25819 consensus_queue.cc:237] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:08.862162 25333 catalog_manager.cc:5582] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb99619485548efa655fee15236b25a (127.19.27.66). New cstate: current_term: 1 leader_uuid: "4cb99619485548efa655fee15236b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } health_report { overall_health: UNKNOWN } } }
W20250810 14:00:08.939868 25524 tablet.cc:2378] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 14:00:09.013173 25657 tablet.cc:2378] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 14:00:09.013568 25790 tablet.cc:2378] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 14:00:09.052369 25478 raft_consensus.cc:1273] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Refusing update from remote peer 4cb99619485548efa655fee15236b25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 14:00:09.052671 25744 raft_consensus.cc:1273] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Refusing update from remote peer 4cb99619485548efa655fee15236b25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 14:00:09.054001 25824 consensus_queue.cc:1035] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250810 14:00:09.054813 25819 consensus_queue.cc:1035] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 14:00:09.081373 25831 mvcc.cc:204] Tried to move back new op lower bound from 7187801739461881856 to 7187801738662711296. Current Snapshot: MvccSnapshot[applied={T|T < 7187801739461881856}]
I20250810 14:00:09.086884 25832 mvcc.cc:204] Tried to move back new op lower bound from 7187801739461881856 to 7187801738662711296. Current Snapshot: MvccSnapshot[applied={T|T < 7187801739461881856}]
I20250810 14:00:09.092687 25833 mvcc.cc:204] Tried to move back new op lower bound from 7187801739461881856 to 7187801738662711296. Current Snapshot: MvccSnapshot[applied={T|T < 7187801739461881856}]
I20250810 14:00:14.225087 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25301
W20250810 14:00:14.626360 25867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:14.627009 25867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:14.659220 25867 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 14:00:15.167937 25523 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:45779 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:45779: connect: Connection refused (error 111)
W20250810 14:00:15.183305 25789 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:45779 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:45779: connect: Connection refused (error 111)
W20250810 14:00:15.191823 25656 heartbeater.cc:646] Failed to heartbeat to 127.19.27.126:45779 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.27.126:45779: connect: Connection refused (error 111)
W20250810 14:00:16.083747 25867 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.375s user 0.002s sys 0.000s
W20250810 14:00:16.084285 25867 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.376s user 0.005s sys 0.000s
I20250810 14:00:16.234369 25867 minidump.cc:252] Setting minidump size limit to 20M
I20250810 14:00:16.237555 25867 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:16.239089 25867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:16.253109 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:16.253123 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:16.391158 25867 server_base.cc:1047] running on GCE node
W20250810 14:00:16.391403 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:16.392567 25867 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250810 14:00:16.393023 25867 hybrid_clock.cc:648] HybridClock initialized: now 1754834416393001 us; error 40375 us; skew 500 ppm
I20250810 14:00:16.393719 25867 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:16.398033 25867 webserver.cc:489] Webserver started at http://0.0.0.0:40691/ using document root <none> and password file <none>
I20250810 14:00:16.398877 25867 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:16.399106 25867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:16.399531 25867 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 14:00:16.403916 25867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/instance:
uuid: "a959f84abb554308976c89a79b0f1782"
format_stamp: "Formatted at 2025-08-10 14:00:16 on dist-test-slave-k4vt"
I20250810 14:00:16.405021 25867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal/instance:
uuid: "a959f84abb554308976c89a79b0f1782"
format_stamp: "Formatted at 2025-08-10 14:00:16 on dist-test-slave-k4vt"
I20250810 14:00:16.410953 25867 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250810 14:00:16.416119 25912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:16.416999 25867 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250810 14:00:16.417384 25867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
uuid: "a959f84abb554308976c89a79b0f1782"
format_stamp: "Formatted at 2025-08-10 14:00:16 on dist-test-slave-k4vt"
I20250810 14:00:16.417821 25867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:16.551556 25867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:16.553007 25867 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:16.553498 25867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:16.558282 25867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:16.572569 25867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Bootstrap starting.
I20250810 14:00:16.577507 25867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:16.579169 25867 log.cc:826] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:16.583148 25867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: No bootstrap required, opened a new log
I20250810 14:00:16.599496 25867 raft_consensus.cc:357] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER }
I20250810 14:00:16.600092 25867 raft_consensus.cc:383] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:16.600328 25867 raft_consensus.cc:738] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a959f84abb554308976c89a79b0f1782, State: Initialized, Role: FOLLOWER
I20250810 14:00:16.601004 25867 consensus_queue.cc:260] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER }
I20250810 14:00:16.601473 25867 raft_consensus.cc:397] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 14:00:16.601831 25867 raft_consensus.cc:491] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 14:00:16.602183 25867 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:16.606817 25867 raft_consensus.cc:513] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER }
I20250810 14:00:16.607467 25867 leader_election.cc:304] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a959f84abb554308976c89a79b0f1782; no voters:
I20250810 14:00:16.609172 25867 leader_election.cc:290] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 14:00:16.609401 25919 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 14:00:16.611492 25919 raft_consensus.cc:695] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 LEADER]: Becoming Leader. State: Replica: a959f84abb554308976c89a79b0f1782, State: Running, Role: LEADER
I20250810 14:00:16.612471 25919 consensus_queue.cc:237] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER }
I20250810 14:00:16.618414 25921 sys_catalog.cc:455] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a959f84abb554308976c89a79b0f1782. Latest consensus state: current_term: 1 leader_uuid: "a959f84abb554308976c89a79b0f1782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER } }
I20250810 14:00:16.619189 25921 sys_catalog.cc:458] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: This master's current role is: LEADER
I20250810 14:00:16.620811 25920 sys_catalog.cc:455] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a959f84abb554308976c89a79b0f1782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER } }
I20250810 14:00:16.621213 25920 sys_catalog.cc:458] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: This master's current role is: LEADER
I20250810 14:00:16.631510 25867 tablet_replica.cc:331] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: stopping tablet replica
I20250810 14:00:16.632130 25867 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 LEADER]: Raft consensus shutting down.
I20250810 14:00:16.632503 25867 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250810 14:00:16.634423 25867 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250810 14:00:16.634822 25867 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250810 14:00:16.660912 25867 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250810 14:00:17.691992 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25393
I20250810 14:00:17.727231 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25527
I20250810 14:00:17.769886 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25660
I20250810 14:00:17.806529 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:45779
--webserver_interface=127.19.27.126
--webserver_port=38703
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.27.126:45779 with env {}
W20250810 14:00:18.115454 25929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:18.116104 25929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:18.116575 25929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:18.148109 25929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 14:00:18.148475 25929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:18.148739 25929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 14:00:18.149013 25929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 14:00:18.184923 25929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.27.126:45779
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.27.126:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.27.126
--webserver_port=38703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:18.186244 25929 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:18.187891 25929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:18.198158 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:18.199916 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:19.405383 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:19.408182 25937 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1204 milliseconds
I20250810 14:00:19.408290 25929 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 14:00:19.409530 25929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:19.412689 25929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:19.414103 25929 hybrid_clock.cc:648] HybridClock initialized: now 1754834419414062 us; error 60 us; skew 500 ppm
I20250810 14:00:19.414866 25929 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:19.421614 25929 webserver.cc:489] Webserver started at http://127.19.27.126:38703/ using document root <none> and password file <none>
I20250810 14:00:19.422456 25929 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:19.422670 25929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:19.430069 25929 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250810 14:00:19.434458 25945 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:19.435462 25929 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250810 14:00:19.435844 25929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
uuid: "a959f84abb554308976c89a79b0f1782"
format_stamp: "Formatted at 2025-08-10 14:00:16 on dist-test-slave-k4vt"
I20250810 14:00:19.437747 25929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:19.484776 25929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:19.486235 25929 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:19.486661 25929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:19.554440 25929 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.126:45779
I20250810 14:00:19.554517 25996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.126:45779 every 8 connection(s)
I20250810 14:00:19.557260 25929 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb
I20250810 14:00:19.563943 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25929
I20250810 14:00:19.565383 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.65:42565
--local_ip_for_outbound_sockets=127.19.27.65
--tserver_master_addrs=127.19.27.126:45779
--webserver_port=46197
--webserver_interface=127.19.27.65
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 14:00:19.567090 25997 sys_catalog.cc:263] Verifying existing consensus state
I20250810 14:00:19.578799 25997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Bootstrap starting.
I20250810 14:00:19.588568 25997 log.cc:826] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:19.600347 25997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 14:00:19.601097 25997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Bootstrap complete.
I20250810 14:00:19.622047 25997 raft_consensus.cc:357] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:19.622742 25997 raft_consensus.cc:738] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a959f84abb554308976c89a79b0f1782, State: Initialized, Role: FOLLOWER
I20250810 14:00:19.623464 25997 consensus_queue.cc:260] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:19.624012 25997 raft_consensus.cc:397] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 14:00:19.624276 25997 raft_consensus.cc:491] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 14:00:19.624596 25997 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 1 FOLLOWER]: Advancing to term 2
I20250810 14:00:19.628659 25997 raft_consensus.cc:513] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:19.629298 25997 leader_election.cc:304] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a959f84abb554308976c89a79b0f1782; no voters:
I20250810 14:00:19.631503 25997 leader_election.cc:290] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 14:00:19.632082 26001 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 14:00:19.635264 26001 raft_consensus.cc:695] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [term 2 LEADER]: Becoming Leader. State: Replica: a959f84abb554308976c89a79b0f1782, State: Running, Role: LEADER
I20250810 14:00:19.636184 26001 consensus_queue.cc:237] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } }
I20250810 14:00:19.636940 25997 sys_catalog.cc:564] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 14:00:19.647235 26002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a959f84abb554308976c89a79b0f1782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } } }
I20250810 14:00:19.649636 26002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: This master's current role is: LEADER
I20250810 14:00:19.648840 26003 sys_catalog.cc:455] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a959f84abb554308976c89a79b0f1782. Latest consensus state: current_term: 2 leader_uuid: "a959f84abb554308976c89a79b0f1782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959f84abb554308976c89a79b0f1782" member_type: VOTER last_known_addr { host: "127.19.27.126" port: 45779 } } }
I20250810 14:00:19.651831 26003 sys_catalog.cc:458] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782 [sys.catalog]: This master's current role is: LEADER
I20250810 14:00:19.653141 26007 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 14:00:19.665710 26007 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=4cde64c7a73549429c7a441edee9c983]
I20250810 14:00:19.672765 26007 tablet_loader.cc:96] loaded metadata for tablet f57feee9554e42678aededb7914ab7ba (table pre_rebuild [id=4cde64c7a73549429c7a441edee9c983])
I20250810 14:00:19.674438 26007 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 14:00:19.704139 26007 catalog_manager.cc:1349] Generated new cluster ID: e8bc20555fcb4591b0e47e5566568973
I20250810 14:00:19.704442 26007 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 14:00:19.729775 26007 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 14:00:19.731859 26007 catalog_manager.cc:1506] Loading token signing keys...
I20250810 14:00:19.756175 26007 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Generated new TSK 0
I20250810 14:00:19.757377 26007 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 14:00:19.933439 25999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:19.933962 25999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:19.934471 25999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:19.965517 25999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:19.966387 25999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.65
I20250810 14:00:20.001132 25999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.65:42565
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.27.65
--webserver_port=46197
--tserver_master_addrs=127.19.27.126:45779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.65
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:20.002439 25999 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:20.004094 25999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:20.016314 26025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:20.022192 26028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:20.019045 26026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:21.513020 26027 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1490 milliseconds
I20250810 14:00:21.513159 25999 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 14:00:21.514385 25999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:21.517237 25999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:21.518710 25999 hybrid_clock.cc:648] HybridClock initialized: now 1754834421518670 us; error 63 us; skew 500 ppm
I20250810 14:00:21.519544 25999 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:21.526767 25999 webserver.cc:489] Webserver started at http://127.19.27.65:46197/ using document root <none> and password file <none>
I20250810 14:00:21.527734 25999 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:21.528031 25999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:21.536314 25999 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250810 14:00:21.541600 26035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:21.542697 25999 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250810 14:00:21.543030 25999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1"
format_stamp: "Formatted at 2025-08-10 14:00:04 on dist-test-slave-k4vt"
I20250810 14:00:21.545038 25999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:21.609957 25999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:21.612007 25999 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:21.612558 25999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:21.615877 25999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 14:00:21.623430 26042 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 14:00:21.631317 25999 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 14:00:21.631634 25999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250810 14:00:21.632011 25999 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 14:00:21.639317 25999 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 14:00:21.639605 25999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.001s sys 0.003s
I20250810 14:00:21.639979 26042 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Bootstrap starting.
I20250810 14:00:21.815891 25999 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.65:42565
I20250810 14:00:21.816113 26148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.65:42565 every 8 connection(s)
I20250810 14:00:21.820694 25999 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb
I20250810 14:00:21.822232 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 25999
I20250810 14:00:21.824136 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.66:35313
--local_ip_for_outbound_sockets=127.19.27.66
--tserver_master_addrs=127.19.27.126:45779
--webserver_port=43743
--webserver_interface=127.19.27.66
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 14:00:21.889185 26149 heartbeater.cc:344] Connected to a master server at 127.19.27.126:45779
I20250810 14:00:21.889714 26149 heartbeater.cc:461] Registering TS with master...
I20250810 14:00:21.890931 26149 heartbeater.cc:507] Master 127.19.27.126:45779 requested a full tablet report, sending...
I20250810 14:00:21.895839 25962 ts_manager.cc:194] Registered new tserver with Master: d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565)
I20250810 14:00:21.903373 25962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.65:35735
I20250810 14:00:21.973284 26042 log.cc:826] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Log is configured to *not* fsync() on all Append() calls
W20250810 14:00:22.178083 26150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:22.178615 26150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:22.179144 26150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 14:00:22.210491 26150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:22.211355 26150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.66
I20250810 14:00:22.251740 26150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.66:35313
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.27.66
--webserver_port=43743
--tserver_master_addrs=127.19.27.126:45779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.66
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:22.253160 26150 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:22.254782 26150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:22.268275 26160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:22.908191 26149 heartbeater.cc:499] Master 127.19.27.126:45779 was elected leader, sending a full tablet report...
W20250810 14:00:23.669773 26159 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 26150
W20250810 14:00:22.270148 26161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:23.849750 26150 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.581s user 0.503s sys 1.047s
W20250810 14:00:23.850390 26150 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.582s user 0.503s sys 1.047s
W20250810 14:00:23.857833 26165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:23.857976 26150 server_base.cc:1047] running on GCE node
I20250810 14:00:23.859285 26150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:23.861447 26150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:23.862794 26150 hybrid_clock.cc:648] HybridClock initialized: now 1754834423862749 us; error 51 us; skew 500 ppm
I20250810 14:00:23.863572 26150 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:23.870105 26150 webserver.cc:489] Webserver started at http://127.19.27.66:43743/ using document root <none> and password file <none>
I20250810 14:00:23.871053 26150 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:23.871305 26150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:23.879843 26150 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20250810 14:00:23.884999 26170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:23.886276 26150 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250810 14:00:23.886726 26150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
uuid: "4cb99619485548efa655fee15236b25a"
format_stamp: "Formatted at 2025-08-10 14:00:06 on dist-test-slave-k4vt"
I20250810 14:00:23.889101 26150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:23.942839 26150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:23.944470 26150 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:23.944932 26150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:23.948060 26150 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 14:00:23.954433 26177 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 14:00:23.961764 26150 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 14:00:23.962018 26150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250810 14:00:23.962306 26150 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 14:00:23.967178 26150 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 14:00:23.967446 26150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250810 14:00:23.967872 26177 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Bootstrap starting.
I20250810 14:00:24.157120 26150 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.66:35313
I20250810 14:00:24.157374 26283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.66:35313 every 8 connection(s)
I20250810 14:00:24.161063 26150 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb
I20250810 14:00:24.169704 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 26150
I20250810 14:00:24.171623 19565 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
/tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.27.67:35679
--local_ip_for_outbound_sockets=127.19.27.67
--tserver_master_addrs=127.19.27.126:45779
--webserver_port=35125
--webserver_interface=127.19.27.67
--builtin_ntp_servers=127.19.27.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 14:00:24.209789 26284 heartbeater.cc:344] Connected to a master server at 127.19.27.126:45779
I20250810 14:00:24.210291 26284 heartbeater.cc:461] Registering TS with master...
I20250810 14:00:24.211767 26284 heartbeater.cc:507] Master 127.19.27.126:45779 requested a full tablet report, sending...
I20250810 14:00:24.216426 25962 ts_manager.cc:194] Registered new tserver with Master: 4cb99619485548efa655fee15236b25a (127.19.27.66:35313)
I20250810 14:00:24.220109 25962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.66:44431
I20250810 14:00:24.302350 26177 log.cc:826] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Log is configured to *not* fsync() on all Append() calls
W20250810 14:00:24.335978 26145 debug-util.cc:398] Leaking SignalData structure 0x7b08000af2c0 after lost signal to thread 26020
W20250810 14:00:24.337164 26145 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5140 after lost signal to thread 26148
I20250810 14:00:24.668123 26042 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 14:00:24.669409 26042 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Bootstrap complete.
I20250810 14:00:24.671345 26042 ts_tablet_manager.cc:1397] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Time spent bootstrapping tablet: real 3.032s user 2.950s sys 0.064s
W20250810 14:00:24.679697 26288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 14:00:24.680505 26288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 14:00:24.681435 26288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250810 14:00:24.688913 26042 raft_consensus.cc:357] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:24.692266 26042 raft_consensus.cc:738] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d804e4e7c4e9457d86ab55cb9e9b25e1, State: Initialized, Role: FOLLOWER
I20250810 14:00:24.693272 26042 consensus_queue.cc:260] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:24.702826 26042 ts_tablet_manager.cc:1428] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Time spent starting tablet: real 0.031s user 0.025s sys 0.004s
W20250810 14:00:24.748467 26288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 14:00:24.750095 26288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.27.67
I20250810 14:00:24.822885 26288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.27.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.27.67:35679
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.27.67
--webserver_port=35125
--tserver_master_addrs=127.19.27.126:45779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.27.67
--log_dir=/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7501
TSAN enabled
I20250810 14:00:24.824733 26288 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 14:00:24.827046 26288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 14:00:24.844444 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:25.224162 26284 heartbeater.cc:499] Master 127.19.27.126:45779 was elected leader, sending a full tablet report...
I20250810 14:00:26.164141 26301 raft_consensus.cc:491] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 14:00:26.176146 26301 raft_consensus.cc:513] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:26.186005 26301 leader_election.cc:290] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cb99619485548efa655fee15236b25a (127.19.27.66:35313), 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
W20250810 14:00:26.204541 26039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.27.67:35679: connect: Connection refused (error 111)
W20250810 14:00:26.215229 26039 leader_election.cc:336] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679): Network error: Client connection negotiation failed: client connection to 127.19.27.67:35679: connect: Connection refused (error 111)
I20250810 14:00:26.220029 26239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4cb99619485548efa655fee15236b25a" is_pre_election: true
W20250810 14:00:26.248029 26039 leader_election.cc:343] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4cb99619485548efa655fee15236b25a (127.19.27.66:35313): Illegal state: must be running to vote when last-logged opid is not known
I20250810 14:00:26.248636 26039 leader_election.cc:304] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d804e4e7c4e9457d86ab55cb9e9b25e1; no voters: 4cb99619485548efa655fee15236b25a, 85ceea734f87464d8b1cb326b1a6c56f
I20250810 14:00:26.250196 26301 raft_consensus.cc:2747] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250810 14:00:26.246192 26296 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 26288
W20250810 14:00:24.848894 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 14:00:26.333469 26296 kernel_stack_watchdog.cc:198] Thread 26288 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 14:00:26.334966 26288 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.492s user 0.505s sys 0.830s
W20250810 14:00:26.335404 26288 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.493s user 0.506s sys 0.830s
I20250810 14:00:26.343628 26288 server_base.cc:1047] running on GCE node
W20250810 14:00:26.344439 26307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 14:00:26.346019 26288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 14:00:26.349056 26288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 14:00:26.350617 26288 hybrid_clock.cc:648] HybridClock initialized: now 1754834426350564 us; error 43 us; skew 500 ppm
I20250810 14:00:26.351892 26288 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 14:00:26.360868 26288 webserver.cc:489] Webserver started at http://127.19.27.67:35125/ using document root <none> and password file <none>
I20250810 14:00:26.362305 26288 fs_manager.cc:362] Metadata directory not provided
I20250810 14:00:26.362632 26288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 14:00:26.374868 26288 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20250810 14:00:26.381350 26312 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 14:00:26.382848 26288 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250810 14:00:26.383306 26288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
uuid: "85ceea734f87464d8b1cb326b1a6c56f"
format_stamp: "Formatted at 2025-08-10 14:00:08 on dist-test-slave-k4vt"
I20250810 14:00:26.386332 26288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 14:00:26.462088 26288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 14:00:26.464262 26288 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 14:00:26.464856 26288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 14:00:26.467965 26288 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 14:00:26.475049 26319 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 14:00:26.482678 26288 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 14:00:26.482935 26288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250810 14:00:26.483225 26288 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 14:00:26.487987 26288 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 14:00:26.488356 26288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250810 14:00:26.488816 26319 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Bootstrap starting.
I20250810 14:00:26.904322 26319 log.cc:826] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Log is configured to *not* fsync() on all Append() calls
I20250810 14:00:27.334857 26426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.27.67:35679 every 8 connection(s)
I20250810 14:00:27.335425 26288 rpc_server.cc:307] RPC server started. Bound to: 127.19.27.67:35679
I20250810 14:00:27.339386 26288 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb
I20250810 14:00:27.348610 19565 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu as pid 26288
I20250810 14:00:27.414968 26427 heartbeater.cc:344] Connected to a master server at 127.19.27.126:45779
I20250810 14:00:27.415477 26427 heartbeater.cc:461] Registering TS with master...
I20250810 14:00:27.416749 26427 heartbeater.cc:507] Master 127.19.27.126:45779 requested a full tablet report, sending...
I20250810 14:00:27.420953 25962 ts_manager.cc:194] Registered new tserver with Master: 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
I20250810 14:00:27.425381 25962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.27.67:33387
I20250810 14:00:27.429912 19565 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 14:00:27.783283 26177 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 14:00:27.784219 26177 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Bootstrap complete.
I20250810 14:00:27.785532 26177 ts_tablet_manager.cc:1397] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Time spent bootstrapping tablet: real 3.818s user 3.599s sys 0.056s
I20250810 14:00:27.790776 26177 raft_consensus.cc:357] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:27.792930 26177 raft_consensus.cc:738] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb99619485548efa655fee15236b25a, State: Initialized, Role: FOLLOWER
I20250810 14:00:27.793607 26177 consensus_queue.cc:260] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:27.796765 26177 ts_tablet_manager.cc:1428] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20250810 14:00:28.436661 26427 heartbeater.cc:499] Master 127.19.27.126:45779 was elected leader, sending a full tablet report...
I20250810 14:00:28.457468 26440 raft_consensus.cc:491] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 14:00:28.457895 26440 raft_consensus.cc:513] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:28.459285 26440 leader_election.cc:290] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cb99619485548efa655fee15236b25a (127.19.27.66:35313), 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
I20250810 14:00:28.460355 26239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4cb99619485548efa655fee15236b25a" is_pre_election: true
I20250810 14:00:28.461165 26239 raft_consensus.cc:2466] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d804e4e7c4e9457d86ab55cb9e9b25e1 in term 1.
I20250810 14:00:28.463974 26039 leader_election.cc:304] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb99619485548efa655fee15236b25a, d804e4e7c4e9457d86ab55cb9e9b25e1; no voters:
I20250810 14:00:28.464809 26440 raft_consensus.cc:2802] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250810 14:00:28.465154 26440 raft_consensus.cc:491] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 14:00:28.465462 26440 raft_consensus.cc:3058] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Advancing to term 2
I20250810 14:00:28.474557 26440 raft_consensus.cc:513] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:28.477243 26239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4cb99619485548efa655fee15236b25a"
I20250810 14:00:28.477680 26440 leader_election.cc:290] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 election: Requested vote from peers 4cb99619485548efa655fee15236b25a (127.19.27.66:35313), 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679)
I20250810 14:00:28.477782 26239 raft_consensus.cc:3058] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Advancing to term 2
I20250810 14:00:28.473078 26382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "85ceea734f87464d8b1cb326b1a6c56f" is_pre_election: true
I20250810 14:00:28.478224 26381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f57feee9554e42678aededb7914ab7ba" candidate_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "85ceea734f87464d8b1cb326b1a6c56f"
W20250810 14:00:28.484776 26039 leader_election.cc:343] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679): Illegal state: must be running to vote when last-logged opid is not known
I20250810 14:00:28.486929 26239 raft_consensus.cc:2466] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d804e4e7c4e9457d86ab55cb9e9b25e1 in term 2.
I20250810 14:00:28.488013 26039 leader_election.cc:304] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cb99619485548efa655fee15236b25a, d804e4e7c4e9457d86ab55cb9e9b25e1; no voters: 85ceea734f87464d8b1cb326b1a6c56f
I20250810 14:00:28.488649 26440 raft_consensus.cc:2802] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 14:00:28.490167 26440 raft_consensus.cc:695] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 LEADER]: Becoming Leader. State: Replica: d804e4e7c4e9457d86ab55cb9e9b25e1, State: Running, Role: LEADER
I20250810 14:00:28.491151 26440 consensus_queue.cc:237] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:28.500775 25961 catalog_manager.cc:5582] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 reported cstate change: term changed from 0 to 2, leader changed from <none> to d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65), VOTER 4cb99619485548efa655fee15236b25a (127.19.27.66) added, VOTER 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67) added, VOTER d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65) added. New cstate: current_term: 2 leader_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20250810 14:00:28.910732 26239 raft_consensus.cc:1273] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 2 FOLLOWER]: Refusing update from remote peer d804e4e7c4e9457d86ab55cb9e9b25e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250810 14:00:28.912384 26440 consensus_queue.cc:1035] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250810 14:00:28.936726 26039 consensus_peers.cc:489] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 -> Peer 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679): Couldn't send request to peer 85ceea734f87464d8b1cb326b1a6c56f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250810 14:00:28.954416 26104 consensus_queue.cc:237] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } }
I20250810 14:00:28.958205 26239 raft_consensus.cc:1273] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 2 FOLLOWER]: Refusing update from remote peer d804e4e7c4e9457d86ab55cb9e9b25e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250810 14:00:28.959437 26443 consensus_queue.cc:1035] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250810 14:00:28.964980 26440 raft_consensus.cc:2953] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } }
I20250810 14:00:28.966439 26239 raft_consensus.cc:2953] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } }
I20250810 14:00:28.971472 25947 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f57feee9554e42678aededb7914ab7ba with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 14:00:28.974272 25961 catalog_manager.cc:5582] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 reported cstate change: config changed from index -1 to 207, VOTER 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67) evicted. New cstate: current_term: 2 leader_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } health_report { overall_health: HEALTHY } } }
I20250810 14:00:28.987250 26104 consensus_queue.cc:237] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:28.989480 26440 raft_consensus.cc:2953] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 4cb99619485548efa655fee15236b25a (127.19.27.66) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } }
I20250810 14:00:28.995579 26361 tablet_service.cc:1515] Processing DeleteTablet for tablet f57feee9554e42678aededb7914ab7ba with delete_type TABLET_DATA_TOMBSTONED (TS 85ceea734f87464d8b1cb326b1a6c56f not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:51818
I20250810 14:00:28.998281 25947 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f57feee9554e42678aededb7914ab7ba with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250810 14:00:28.999610 25949 catalog_manager.cc:4908] TS 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679): delete failed for tablet f57feee9554e42678aededb7914ab7ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f57feee9554e42678aededb7914ab7ba already in progress: opening tablet
I20250810 14:00:29.002120 25961 catalog_manager.cc:5582] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 reported cstate change: config changed from index 207 to 208, VOTER 4cb99619485548efa655fee15236b25a (127.19.27.66) evicted. New cstate: current_term: 2 leader_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } health_report { overall_health: HEALTHY } } }
I20250810 14:00:29.019847 26219 tablet_service.cc:1515] Processing DeleteTablet for tablet f57feee9554e42678aededb7914ab7ba with delete_type TABLET_DATA_TOMBSTONED (TS 4cb99619485548efa655fee15236b25a not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:35142
I20250810 14:00:29.022497 26459 tablet_replica.cc:331] stopping tablet replica
I20250810 14:00:29.023087 26459 raft_consensus.cc:2241] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250810 14:00:29.023583 26459 raft_consensus.cc:2270] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250810 14:00:29.044466 26459 ts_tablet_manager.cc:1905] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 14:00:29.056558 26459 ts_tablet_manager.cc:1918] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250810 14:00:29.057057 26459 log.cc:1199] T f57feee9554e42678aededb7914ab7ba P 4cb99619485548efa655fee15236b25a: Deleting WAL directory at /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/wal/wals/f57feee9554e42678aededb7914ab7ba
I20250810 14:00:29.058879 25949 catalog_manager.cc:4928] TS 4cb99619485548efa655fee15236b25a (127.19.27.66:35313): tablet f57feee9554e42678aededb7914ab7ba (table pre_rebuild [id=4cde64c7a73549429c7a441edee9c983]) successfully deleted
I20250810 14:00:29.232539 26319 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 14:00:29.233568 26319 tablet_bootstrap.cc:492] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Bootstrap complete.
I20250810 14:00:29.235321 26319 ts_tablet_manager.cc:1397] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Time spent bootstrapping tablet: real 2.747s user 2.631s sys 0.067s
I20250810 14:00:29.242731 26319 raft_consensus.cc:357] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:29.245608 26319 raft_consensus.cc:738] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ceea734f87464d8b1cb326b1a6c56f, State: Initialized, Role: FOLLOWER
I20250810 14:00:29.246426 26319 consensus_queue.cc:260] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } }
I20250810 14:00:29.249950 26319 ts_tablet_manager.cc:1428] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20250810 14:00:29.254528 26361 tablet_service.cc:1515] Processing DeleteTablet for tablet f57feee9554e42678aededb7914ab7ba with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:51818
I20250810 14:00:29.255733 26457 tablet_replica.cc:331] stopping tablet replica
I20250810 14:00:29.256568 26457 raft_consensus.cc:2241] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250810 14:00:29.257300 26457 raft_consensus.cc:2270] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250810 14:00:29.281773 26457 ts_tablet_manager.cc:1905] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 14:00:29.298887 26457 ts_tablet_manager.cc:1918] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250810 14:00:29.299291 26457 log.cc:1199] T f57feee9554e42678aededb7914ab7ba P 85ceea734f87464d8b1cb326b1a6c56f: Deleting WAL directory at /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/wal/wals/f57feee9554e42678aededb7914ab7ba
I20250810 14:00:29.300885 25949 catalog_manager.cc:4928] TS 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679): tablet f57feee9554e42678aededb7914ab7ba (table pre_rebuild [id=4cde64c7a73549429c7a441edee9c983]) successfully deleted
W20250810 14:00:29.362180 19565 scanner-internal.cc:458] Time spent opening tablet: real 1.852s user 0.004s sys 0.004s
I20250810 14:00:29.991897 26219 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 14:00:29.992005 26361 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 14:00:30.006706 26084 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
a959f84abb554308976c89a79b0f1782 | 127.19.27.126:45779 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:33783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4cb99619485548efa655fee15236b25a | 127.19.27.66:35313 | HEALTHY | <none> | 0 | 0
85ceea734f87464d8b1cb326b1a6c56f | 127.19.27.67:35679 | HEALTHY | <none> | 0 | 0
d804e4e7c4e9457d86ab55cb9e9b25e1 | 127.19.27.65:42565 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.19.27.65 | experimental | 127.19.27.65:42565
local_ip_for_outbound_sockets | 127.19.27.66 | experimental | 127.19.27.66:35313
local_ip_for_outbound_sockets | 127.19.27.67 | experimental | 127.19.27.67:35679
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb | hidden | 127.19.27.65:42565
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb | hidden | 127.19.27.66:35313
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb | hidden | 127.19.27.67:35679
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:33783 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250810 14:00:30.294759 19565 log_verifier.cc:126] Checking tablet f57feee9554e42678aededb7914ab7ba
I20250810 14:00:30.599294 19565 log_verifier.cc:177] Verified matching terms for 208 ops in tablet f57feee9554e42678aededb7914ab7ba
I20250810 14:00:30.601563 25962 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34862:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250810 14:00:30.602028 25962 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34862:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250810 14:00:30.613092 25962 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a959f84abb554308976c89a79b0f1782: Sending DeleteTablet for 1 replicas of tablet f57feee9554e42678aededb7914ab7ba
I20250810 14:00:30.615024 19565 test_util.cc:276] Using random seed: 987782616
I20250810 14:00:30.614918 26084 tablet_service.cc:1515] Processing DeleteTablet for tablet f57feee9554e42678aededb7914ab7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-10 14:00:30 UTC) from {username='slave'} at 127.0.0.1:44492
I20250810 14:00:30.616890 26496 tablet_replica.cc:331] stopping tablet replica
I20250810 14:00:30.617645 26496 raft_consensus.cc:2241] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 LEADER]: Raft consensus shutting down.
I20250810 14:00:30.618453 26496 raft_consensus.cc:2270] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250810 14:00:30.637531 25962 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39002:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250810 14:00:30.640069 25962 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 14:00:30.661569 26361 tablet_service.cc:1468] Processing CreateTablet for tablet 73073286280944e2a349f6526b6d6054 (DEFAULT_TABLE table=post_rebuild [id=24200790805c434d9e2010f8e6c6c281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 14:00:30.662375 26496 ts_tablet_manager.cc:1905] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250810 14:00:30.662925 26361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73073286280944e2a349f6526b6d6054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:30.663002 26219 tablet_service.cc:1468] Processing CreateTablet for tablet 73073286280944e2a349f6526b6d6054 (DEFAULT_TABLE table=post_rebuild [id=24200790805c434d9e2010f8e6c6c281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 14:00:30.664300 26219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73073286280944e2a349f6526b6d6054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:30.674279 26084 tablet_service.cc:1468] Processing CreateTablet for tablet 73073286280944e2a349f6526b6d6054 (DEFAULT_TABLE table=post_rebuild [id=24200790805c434d9e2010f8e6c6c281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 14:00:30.675717 26084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73073286280944e2a349f6526b6d6054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 14:00:30.679610 26496 ts_tablet_manager.cc:1918] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250810 14:00:30.680157 26496 log.cc:1199] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Deleting WAL directory at /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/wal/wals/f57feee9554e42678aededb7914ab7ba
I20250810 14:00:30.681355 26496 ts_tablet_manager.cc:1939] T f57feee9554e42678aededb7914ab7ba P d804e4e7c4e9457d86ab55cb9e9b25e1: Deleting consensus metadata
I20250810 14:00:30.684962 26503 tablet_bootstrap.cc:492] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f: Bootstrap starting.
I20250810 14:00:30.690553 25947 catalog_manager.cc:4928] TS d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565): tablet f57feee9554e42678aededb7914ab7ba (table pre_rebuild [id=4cde64c7a73549429c7a441edee9c983]) successfully deleted
I20250810 14:00:30.692582 26503 tablet_bootstrap.cc:654] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:30.699182 26504 tablet_bootstrap.cc:492] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1: Bootstrap starting.
I20250810 14:00:30.702389 26505 tablet_bootstrap.cc:492] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a: Bootstrap starting.
I20250810 14:00:30.704001 26503 tablet_bootstrap.cc:492] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f: No bootstrap required, opened a new log
I20250810 14:00:30.704459 26503 ts_tablet_manager.cc:1397] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20250810 14:00:30.707100 26503 raft_consensus.cc:357] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.707823 26503 raft_consensus.cc:383] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:30.708156 26503 raft_consensus.cc:738] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ceea734f87464d8b1cb326b1a6c56f, State: Initialized, Role: FOLLOWER
I20250810 14:00:30.709025 26505 tablet_bootstrap.cc:654] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:30.708791 26503 consensus_queue.cc:260] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.710965 26504 tablet_bootstrap.cc:654] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1: Neither blocks nor log segments found. Creating new log.
I20250810 14:00:30.717098 26505 tablet_bootstrap.cc:492] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a: No bootstrap required, opened a new log
I20250810 14:00:30.717581 26505 ts_tablet_manager.cc:1397] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20250810 14:00:30.720475 26505 raft_consensus.cc:357] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.723239 26503 ts_tablet_manager.cc:1428] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20250810 14:00:30.724382 26505 raft_consensus.cc:383] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:30.724773 26505 raft_consensus.cc:738] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb99619485548efa655fee15236b25a, State: Initialized, Role: FOLLOWER
I20250810 14:00:30.725785 26504 tablet_bootstrap.cc:492] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1: No bootstrap required, opened a new log
I20250810 14:00:30.725605 26505 consensus_queue.cc:260] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.726188 26504 ts_tablet_manager.cc:1397] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.000s
I20250810 14:00:30.728821 26504 raft_consensus.cc:357] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.729470 26504 raft_consensus.cc:383] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 14:00:30.729758 26504 raft_consensus.cc:738] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d804e4e7c4e9457d86ab55cb9e9b25e1, State: Initialized, Role: FOLLOWER
I20250810 14:00:30.730373 26504 consensus_queue.cc:260] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.736483 26505 ts_tablet_manager.cc:1428] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a: Time spent starting tablet: real 0.019s user 0.009s sys 0.003s
I20250810 14:00:30.738003 26504 ts_tablet_manager.cc:1428] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250810 14:00:30.825027 26510 raft_consensus.cc:491] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 14:00:30.825539 26510 raft_consensus.cc:513] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.828573 26510 leader_election.cc:290] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679), d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565)
I20250810 14:00:30.854403 26382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73073286280944e2a349f6526b6d6054" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ceea734f87464d8b1cb326b1a6c56f" is_pre_election: true
I20250810 14:00:30.855147 26382 raft_consensus.cc:2466] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 0.
I20250810 14:00:30.855993 26104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73073286280944e2a349f6526b6d6054" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" is_pre_election: true
I20250810 14:00:30.856395 26174 leader_election.cc:304] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb99619485548efa655fee15236b25a, 85ceea734f87464d8b1cb326b1a6c56f; no voters:
I20250810 14:00:30.856703 26104 raft_consensus.cc:2466] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 0.
I20250810 14:00:30.857263 26510 raft_consensus.cc:2802] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 14:00:30.857631 26510 raft_consensus.cc:491] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 14:00:30.857969 26510 raft_consensus.cc:3058] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:30.864584 26510 raft_consensus.cc:513] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
I20250810 14:00:30.866977 26382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73073286280944e2a349f6526b6d6054" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ceea734f87464d8b1cb326b1a6c56f"
I20250810 14:00:30.867132 26104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73073286280944e2a349f6526b6d6054" candidate_uuid: "4cb99619485548efa655fee15236b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1"
I20250810 14:00:30.866600 26510 leader_election.cc:290] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 election: Requested vote from peers 85ceea734f87464d8b1cb326b1a6c56f (127.19.27.67:35679), d804e4e7c4e9457d86ab55cb9e9b25e1 (127.19.27.65:42565)
I20250810 14:00:30.867496 26382 raft_consensus.cc:3058] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:30.867550 26104 raft_consensus.cc:3058] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 0 FOLLOWER]: Advancing to term 1
I20250810 14:00:30.872601 26104 raft_consensus.cc:2466] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 1.
I20250810 14:00:30.873624 26172 leader_election.cc:304] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb99619485548efa655fee15236b25a, d804e4e7c4e9457d86ab55cb9e9b25e1; no voters:
I20250810 14:00:30.874115 26382 raft_consensus.cc:2466] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb99619485548efa655fee15236b25a in term 1.
I20250810 14:00:30.874246 26510 raft_consensus.cc:2802] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 1 FOLLOWER]: Leader election won for term 1
I20250810 14:00:30.875970 26510 raft_consensus.cc:695] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [term 1 LEADER]: Becoming Leader. State: Replica: 4cb99619485548efa655fee15236b25a, State: Running, Role: LEADER
I20250810 14:00:30.876812 26510 consensus_queue.cc:237] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } }
W20250810 14:00:30.877885 26151 tablet.cc:2378] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 14:00:30.885049 25962 catalog_manager.cc:5582] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb99619485548efa655fee15236b25a (127.19.27.66). New cstate: current_term: 1 leader_uuid: "4cb99619485548efa655fee15236b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cb99619485548efa655fee15236b25a" member_type: VOTER last_known_addr { host: "127.19.27.66" port: 35313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 } health_report { overall_health: UNKNOWN } } }
W20250810 14:00:30.891717 26432 tablet.cc:2378] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 14:00:30.936890 26285 tablet.cc:2378] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 14:00:31.092216 26104 raft_consensus.cc:1273] T 73073286280944e2a349f6526b6d6054 P d804e4e7c4e9457d86ab55cb9e9b25e1 [term 1 FOLLOWER]: Refusing update from remote peer 4cb99619485548efa655fee15236b25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 14:00:31.092303 26382 raft_consensus.cc:1273] T 73073286280944e2a349f6526b6d6054 P 85ceea734f87464d8b1cb326b1a6c56f [term 1 FOLLOWER]: Refusing update from remote peer 4cb99619485548efa655fee15236b25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 14:00:31.093966 26516 consensus_queue.cc:1035] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ceea734f87464d8b1cb326b1a6c56f" member_type: VOTER last_known_addr { host: "127.19.27.67" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250810 14:00:31.094739 26510 consensus_queue.cc:1035] T 73073286280944e2a349f6526b6d6054 P 4cb99619485548efa655fee15236b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d804e4e7c4e9457d86ab55cb9e9b25e1" member_type: VOTER last_known_addr { host: "127.19.27.65" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 14:00:31.131768 26524 mvcc.cc:204] Tried to move back new op lower bound from 7187801829737664512 to 7187801828879421440. Current Snapshot: MvccSnapshot[applied={T|T < 7187801829737664512}]
W20250810 14:00:31.659996 26519 meta_cache.cc:1261] Time spent looking up entry by key: real 0.080s user 0.001s sys 0.003s
I20250810 14:00:35.956467 26219 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 14:00:35.957635 26361 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 14:00:35.960491 26084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
a959f84abb554308976c89a79b0f1782 | 127.19.27.126:45779 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:33783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4cb99619485548efa655fee15236b25a | 127.19.27.66:35313 | HEALTHY | <none> | 1 | 0
85ceea734f87464d8b1cb326b1a6c56f | 127.19.27.67:35679 | HEALTHY | <none> | 0 | 0
d804e4e7c4e9457d86ab55cb9e9b25e1 | 127.19.27.65:42565 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.19.27.65 | experimental | 127.19.27.65:42565
local_ip_for_outbound_sockets | 127.19.27.66 | experimental | 127.19.27.66:35313
local_ip_for_outbound_sockets | 127.19.27.67 | experimental | 127.19.27.67:35679
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-0/data/info.pb | hidden | 127.19.27.65:42565
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-1/data/info.pb | hidden | 127.19.27.66:35313
server_dump_info_path | /tmp/dist-test-taskW7yu7w/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754834243974047-19565-0/minicluster-data/ts-2/data/info.pb | hidden | 127.19.27.67:35679
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.27.84:33783 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250810 14:00:36.148267 19565 log_verifier.cc:126] Checking tablet 73073286280944e2a349f6526b6d6054
I20250810 14:00:36.923491 19565 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 73073286280944e2a349f6526b6d6054
I20250810 14:00:36.924546 19565 log_verifier.cc:126] Checking tablet f57feee9554e42678aededb7914ab7ba
I20250810 14:00:36.924813 19565 log_verifier.cc:177] Verified matching terms for 0 ops in tablet f57feee9554e42678aededb7914ab7ba
I20250810 14:00:36.949540 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25999
I20250810 14:00:36.997888 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 26150
I20250810 14:00:37.041745 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 26288
I20250810 14:00:37.076227 19565 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskW7yu7w/build/tsan/bin/kudu with pid 25929
2025-08-10T14:00:37Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36620 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36620 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (193094 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST